Keywords

Key Concepts

  • The dramatic changes in healthcare and surgical training have forced educators to adopt a new approach with an emphasis on outcomes and competence.

  • A major impediment to surgical education is the lack of hospital or administrative support . The fiscal solvency of most academic institutions is more dependent on a high volume, cost-effective, and efficient department of surgery than ever before.

  • Work hour restrictions are another major impediment to training. Residents must now make the most of all learning opportunities, as learners no longer have the “luxury” of unlimited clinical immersion.

  • Simulation and surgical skills laboratories will, over time, have an increasing role in training for general technical skills, such as knot tying and procedure-specific skills. The successful application for procedure-specific training with virtual reality systems has recently been demonstrated in several trials.

  • Competency-based medical education (CBME ) is “an approach to preparing physicians for practice that is fundamentally oriented to graduated outcome abilities and organized around competencies derived from an analysis of societal and patient needs.”

  • To meet ACGME requirements , multiple assessments will be required to ensure milestone progression along all of the competencies. Assessments that are commonly used include the In-Training Evaluation Report (ITER), 360-Degree evaluations (including patient surveys), chart stimulated recall, oral examinations, multiple choice examinations, portfolios, and simulations and models.

  • Post-graduate medical education now demands an increase in support at a time of waning resource allocation and protected time. In this climate, education often takes a backseat.

Introduction

Major changes in the way we train surgeons are occurring in several areas. One of the developments in medical education over the past decade which will have the largest impact on graduate education has been a shift in the focus from the processes of education to the outcomes of education, or development of competencies and attainment of milestones [1, 2]. This focus on outcome assessment and milestone achievement will require that training programs make better use of other advances in surgical education, such as simulation. Simulation-based training has also been used to help accelerate learners’ growth in knowledge, skills, and attitudes, prior to entering the clinical arena, and the literature evaluating the use of simulation-based learning will be examined. As well, a competency-based framework requires that multiple formative and summative assessments be used to ensure that the required outcomes are achieved. To help in assessing the outcomes of training, practical methods that are available to assess learners in the six broad competencies of the Accreditation Council for Graduate Medical Education (ACGME) outcome project will be discussed. Finally, we will use examples from the Colorectal Surgery Milestones from the American Board of Colorectal Surgery and ACGME milestones project to demonstrate how one can integrate simulation and evaluation into a colorectal residency to help ensure all residents achieve milestones in a timely fashion.

Challenges to Surgical Education

Surgical education has seen many challenges over the past 2 decades. Surgical care is in an era of increased emphasis on accountability and outcomes [3]. Physicians need to be better trained to weigh the cost and value of diagnostic and therapeutic interventions, as there is more focus on cost containment [4]. At the same time, there is a stress on patient safety and error mitigation [5]. Work hour restrictions require that residents must make the most of all learning opportunities, as learners no longer have the “luxury” of unlimited clinical immersion [6]. With fewer duty hours, some learning has moved away from the clinical environment to simulated environments to ensure learning objectives are met, and so that residents can make the most of learning opportunities presented to them in the limited hours available [7]. All of these factors impact on the ability to rely on the traditional Halstedian model of graduate medical education, whereby learners would iteratively learn to deal with most surgical problems through stepwise progression in training, enhanced by a large volume of exposure. Although this model served surgical education quite well for the past century, many would argue that it led to deficits in knowledge for practitioners, with “lacunae” of knowledge, or skills deficits. Because passing through the system was based on meeting an overall minimum standard, surgeons could graduate without adequate exposure or expertise in some clinical areas. Furthermore, insufficient attention to areas such as quality improvement, the use of new technologies, and ability to respond to shifting patient expectations, patient demographics, and health care delivery systems led to practitioners potentially unable to adequately serve the needs of the population [3].

These challenges have created a monumental task for trainees and educators that must be supported from the top down. Contemporary post-graduate medical education demands an increase in support at a time of waning resource allocation and protected time. As we move away from a Halstedian model, and try to make more of each teaching hour available, the teaching demands on faculty increase. Administration must recognize this when allocating resources, or resident teaching will suffer. At each academic institution, the vision of education imparted from the department chair and or administration must be articulated and clearly outlined. Unfortunately, allocating additional time for surgeons to teach residents, both inside and outside the operating room, has not been a major priority for administration, but rather, almost a foreign concept. This is, and will continue to be, a major impediment to training the surgeons of the future. More recently, several institutions have also exchanged their salary-based system for faculty with an incentive-driven compensation plan for increased volumes. This alteration may further impact on resident training. In essence, academic surgeons are being asked to do more with less clinically, while maintaining their research interests and training responsibilities. These changes have the potential to foster a pessimistic and apathetic attitude amongst academic surgeons in regards to training, however, we must recognize the critical role that our residents play in patient care. Without residents, patient volumes would diminish, our academic aspirations would wane as we would have even less protected time, and our lifestyles would dramatically change. Academic surgeons should ideally have their clinical volumes evaluated as a 90 % FTE (Full Time Employee) with 10 % allotted for teaching. This argument can be strengthened by numerous publications demonstrating that cases with trainees take significantly longer [8]. Without this fundamental change, many may argue that they are not being paid to teach. This may be true in principle, but faculty who feel they derive no benefit from residents should take their own patient calls on the floor, do consults in the emergency room, enter orders on patients, and take over all of the other duties residents perform. Irrespective of the frustrations that exist with administration, the relationship of the trainer and trainee is give and take. Trainees are also often frustrated with the system, and in general, are trying to become the best surgeons possible. Remembering that the deficiencies in reimbursement, time commitment, and resources allocation lie more with the system than with the residents can help faculty to deal with some of the frustrations inherent in teaching.

Competency-Based Medical Education

A fundamental change in residency may be required to deal with many of the changes that have evolved. A major focus of new models in post-graduate medical education is a focus on demonstration of attainment of competency, with a shift away from time and objective-based training to a competency-based framework. Examples of this change in culture can be appreciated in different Health Care Systems. The Outcome Project of the ACGME in the USA [1] and CanMEDS Competency by Design in Canada [2] focus on outcomes and abilities of the learner, with explicit competencies as the organizing principle of curricular design. With the Next Accreditation System (NAS ) of the ACGME, and Competency by Design within the Royal College, the focus of accreditation will move away from assessing the objectives and processes of education, and towards assessing the outcomes of education. Programs will have to demonstrate that their trainees have acquired the knowledge, skills, and attitudes of the competencies required for safe and effective practice. Three concepts that have been forwarded in this move towards competency-based education are competencies, milestones, and entrustable professional activities (EPAs). These build upon each other.

Competency-based medical education (CBME) is “an approach to preparing physicians for practice that is fundamentally oriented to graduate outcome abilities and organized around competencies derived from an analysis of societal and patient needs. It de-emphasizes time-based training and promises greater accountability, flexibility and learner-centeredness” [9].

Competencies integrate knowledge, skills, and attitudes. They are observable and can be measured. Ideally, learners will demonstrate progressive attainment of competencies as they move from novice to expert, assembling competencies like building blocks as they develop knowledge, skills, and attitudes. These building blocks are the stepping stones to milestone attainment. By explicitly identifying milestones that learners should achieve, competency-based education helps move learners along the pathway to excellence. A milestone is a defined, observable marker of an individual’s ability along a developmental continuum [1]. Milestones are useful for planning and teaching, as frequent assessment of the milestone attainment of residents allows the program to assess where the learner is, where deficits might be, and plan learning opportunities accordingly. Although a competency-based curriculum is generally planned in terms of milestones, the eventual goal of a post-graduate training program is to ensure that graduates can carry out the essential tasks of the specialty. An Entrustable Professional Activity (EPA ) is an essential task of a “discipline” that an individual can be trusted to perform independently in a given context [10]. Based on the demonstration of sufficient competence, the supervisor feels the resident is able to do the task independently. EPAs can be used for overall assessment and decisions as to when residents are ready for independent practice. Typically an EPA integrates multiple milestones. EPAs are the tasks or activities that must be accomplished (for example, manage a patient with rectal cancer), whereas milestones are the abilities of the individual on a continuum (for example, able to “assess imaging information and justify a TNM-based treatment strategy”).

A central tenet of CBME is that learners must assume greater responsibility for their own learning and the assessment of their learning than in traditional approaches. Learners become responsible for developing a learning portfolio, and collecting formative and summative assessments to demonstrate they have achieved the desired outcomes. Thus, the competency-based framework places more onus for ownership of learning on the trainee, who must ensure they are actively seeking experiences to enable them to achieve milestones. They must also demonstrate, through assessment tools, their achievement. By putting more onus on the learner to take charge of their own development during post-graduate medical education, and focusing on the need to seek out learning opportunities and ensure milestones are reached, the hope is that competency-based education will better ensure that learners have the skills needed for lifelong self-directed learning and continuing professional development as they move along the medical education continuum into independent practice.

Strategies Outside of the Operating Room

In addition to increased accountability, the current paradigm of surgical education also advocates non-clinical, or ex-vivo, methods of training to improve clinical performance by providing practice opportunity in a safe environment. Surgical educators uniformly agree that technical skills exercises and training in the surgical skills center, designed specifically to allow the resident to optimize their operative learning and experience, will play a critical role. The American Board of Surgeons Resident Review Committee has made it mandatory that all surgical training programs have a means of training outside the operating room [11]. Therefore, simulation and surgical skills laboratories will, over time, have an increasing role in training for general technical skills, such as knot tying and procedure-specific skills (Figure 69-1). In its broadest terms, simulation is defined as the act of imitating the behavior of some situation or some process by means of something suitably analogous. Therefore, the majority of non-clinical technical skill exercises, regardless of the model, qualify as “simulation.” Current platforms vary considerably in level of fidelity, from box trainers to technologically advanced Virtual Reality (VR) programs.

Figure 69-1.
figure 1

Pelvic Pouch Skills Lab Station at the Institute for Surgery and Innovation at Case Medical Center, Cleveland, OH.

Full size image

Simple box trainers for laparoscopic skills such as the validated MISTELS (McGill Inanimate System for Training and Evaluation of Laparoscopic Skills) are effective at the junior trainee level and should be readily incorporated into any laboratory curricula. VR platforms have also been shown to improve performance in the operating room. More specifically, dedicated practice with VR simulators have correlated with improved operative times, and efficiency of movement for clinical laparoscopic cholecystectomy . Seymour et al. evaluated 16 residents of varying levels and compared clinical laparoscopic cholecystectomy outcomes between residents who received training on a VR system versus those who did not. They found no difference in baseline assessments between the two groups, but found that residents who trained on the simulator were faster, made fewer errors, and were less likely to injure the gallbladder in the operating room [12]. Grantcharov et al. also evaluated 16 residents and compared training on a VR simulator to a control group. They found improved economy of movements and fewer errors in residents who were trained on a VR simulator [13]. Beyer and colleagues evaluated transfer of skills from simulators to the operating room using the Global Operative Assessment of Laparoscopic Skills (GOALS), a validated laparoscopic skills assessment model [14]. Their prospective trial involving 19 residents found improved GOAL scores in residents who were trained on a simulator when compared to those who were not.

Simbionix (Cleveland, OH) offers a VR (LAP Mentor) model for laparoscopic sigmoid colectomy that more accurately portrays resection in the operating room than previous hybrid systems [15] (Figure 69-2). Evidence of face, content, and construct validity have been established for general procedures with the LAP Mentor VR system [16]. More recently, evidence of construct validity was established for certain metrics, specifically with the laparoscopic sigmoid model [17]. In this study, the metrics assessing the instrument path length, the accuracy of the medial peritoneal mobilization, and the quality of the IMA dissection demonstrated the strongest ability to discriminate between general surgeons and expert colorectal surgeons. However, construct validity was not established for technical errors, as the model could not differentiate between experts and novice surgeons using this metric.

Figure 69-2.
figure 2

Virtual reality laparoscopic sigmoid module (Simbionix, Cleveland, OH).

Full size image

The successful application for procedure-specific training with VR systems has recently been demonstrated in several trials. Calatayud et al. tested “warm up” with a VR system prior to laparoscopic cholecystectomy and found that Objective Structured Assessment of Technical Skill (OSATS) global rating scales were better after practice [18]. Palter and Grantcharov developed a comprehensive ex-vivo pre-operative training curriculum that improved performance for LC [19]. This study involved preparation with simulation, cognitive training, and participation in a cadaver lab. In this study, residents were PGY-2 through 4, having previously completed FLS and possessing some advanced laparoscopic experience. Using an entire curriculum that addressed multiple aspects of performance, which included procedure-specific simulation, overall laparoscopic colectomy skills were enhanced. While this impressive approach was successful, having all trainees perform this labor intense program prior to operative procedures may not be practical. The cost and time requirements of this model are likely not likely sustainable in most training programs outside of a trial. However, this well-designed trial most importantly demonstrates that preparation can improve performance for laparoscopic colon resection. More recently, Singh et al. [20] utilized a validated virtual reality laparoscopic cholecystectomy curriculum to study the role of video-based coaching in teaching laparoscopic skills. The authors found that video-based coaching enhanced the quality of laparoscopic performance on both virtual reality and porcine models. Simulation curriculums for endoscopic procedures have also been studied and found to be effective. Williams et al. compared general surgery residents to gastroenterologists in their ability to perform colonoscopy after the trainees completed an endoscopic simulation-training curriculum. They discovered that the trainees were capable of achieving quality measures equivalent to faculty gastroenterologists [21]. Furthermore, Iordache et al. recently validated a cadaver model with simulated training to place endoscopic colonic stents. They found that the model had reliability and evidence for construct validity [22].

Each academic institution may choose to incorporate one of these preparation models or a variation on this theme. The VR studies for basic laparoscopic skills training have routinely incorporated a proficiency-based model, whereby trainees have proficiency targets to meet, rather than time on task as a training goal [23].

Understanding Competency-Based Medical Education

Traditionally, residency has been primarily time-based, with time spent on rotations used as a surrogate for competence. In a pure CBME model, demonstration of defined competence in a time-free model would be used. This time-free approach is usually not practical, as rotations need to be somewhat structured, and learners gain competencies at very different rates. Thus, a hybrid model, in which learners move through rotations and other structured learning approaches (such as simulation-based learning), but only graduate once milestones are met, is advocated. Summative assessments are used to ensure competencies are attained, and residents are progressing satisfactorily with milestone achievement. Teachers ensure learners have the necessary learning opportunities, and use real time direct observation to evaluate achievement.

Carraccio [24] has compared traditional time-dependent models of curriculum development to pure competency-based curricula, which are time free (Table 69-1). In a time-dependent model, the main organizing structure would be time spent on rotations, as opposed to progression of competence. Rotations, with academic half days and formal teaching are the main structures of this type of curriculum, with the learning goals being predefined objectives of training. In-training assessments are used to ensure that rotations are passed, but are usually completed at the end of a pre-defined time. Teacher’s roles are primarily supervision and teaching, and learners provide service, attend academic sessions, and study for exams. In a time-free model, progression of competence, rather than rotations is the main organizing structure, with rotations seen as one of many resources to aid learning. Milestones are the learning goals, and the role of timed rotations is irrelevant. Assessment is focused on documentation of milestone achievement, with summative assessment used to ensure attainment of competencies. Teachers supervise, teach and also directly observe in clinical settings to ensure competencies are achieved. Learners must take ownership of their learning, plotting a course for progression of competence through a variety of learning activities. Once the competencies have been demonstrated to have been met, learners move on. Realistically, most programs will use a hybrid of these extremes, as development of a pure competency-based model is difficult to structure and monitor.

Table 69-1. Comparison of competency-based education and traditional curricular models
Full size table

Some of the major challenges when thinking of implementing CBME are similar across specialties. In Internal Medicine , three of the major challenges identified were: incorporating practice-based learning and improvement and systems-based practice into the curriculum; evaluating residents across the competencies; and ensuring advancement based on competence, rather than time [25]. All of these will be major challenges for colorectal programs , which are short compared to most residencies, especially as the incoming residents may already require some degree of remediation [26]. However, practice-based learning and improvement can be incorporated into the everyday clinical context. For example, residents can be involved in quality initiatives, identifying and pursuing improvement processes, structured morbidity and mortality conferences, and provided with opportunities to identify gaps or improve measures. Furthermore, systems-based practice can be best addressed during transitions in patient care, from inpatient to outpatient and other settings.

Evaluating residents across all of the competencies is also challenging. A discussion of assessment modalities follows, but in general, direct observation in the clinical setting will likely be the most feasible method of assessment for a small program, as more structured assessment tools, such as standardized patient encounters, and performance-based assessments can be expensive and difficult to administer for only a few residents. The challenge for programs is to ensure they are documenting enough evaluative feedback to ensure milestone assessment is reliable and valid.

Assessment of Performance

Assessment of performance during post-graduate training can be either formative (meant to give feedback, or direction to learners) or summative (high stakes, end-of-training, such as certification decisions). Within a CBME framework, assessment of learning becomes vital to the entire process, and the assessment framework must be robust enough to ensure that developmental milestones are accurately assessed. Formative assessment is done frequently, to enhance reliability and validity, and is based around real clinical work. This allows for performance to be assessed across all of the competencies.

In a CBME model, the assessment process must become “more learner driven, learner focused and formative” [26]. Multiple types of assessment tools are available to help the Clinical Competency Committee (CCC) [27] make valid judgments about the milestone progress of their learners. To meet ACGME requirements, multiple assessments will be required to ensure milestone progression along all of the competencies. Programs will have to use a variety of assessments to provide their CCC with robust data.

Assessments that are commonly used include the In-Training Evaluation Report (ITER), 360-Degree evaluations (including patient surveys), chart stimulated recall, oral examinations, multiple choice examinations, portfolios, and simulations and models. Table 69-2 outlines these commonly used tools, and what competencies are best evaluated through their use. Evaluations used specifically for technical skill assessment in colorectal surgery include Performance Based Assessments (PBAs), operation-specific rating scales (e.g., those used for laparoscopic colectomy), outcome measures (e.g., cecal intubation rates), and final summative assessments [e.g., the Colorectal Objective Structured Assessment of Technical Skills (COSATS) ]. Each of these evaluation tools will be briefly discussed.

Table 69-2. Evaluating outcomes in the core competencies
Full size table

In-Training Evaluation Report

Assessment during post-graduate training has traditionally relied on the ITER. The ITER has the positive attributes of theoretically being comprehensive, being able to assess across all of the competencies, and being relatively easy to use, even in small programs, such as colorectal surgery. However, although in theory ITERs have the potential to give feedback on real world performance, the way in which they are operationalized in many programs make them suboptimal. Often, they are completed long after the training period, when recall may not be ideal. Assessors often use the “above average” portion of the form almost exclusively, and are reluctant to fail residents. Because of the failure to use the range of marks, the reliability of ITERs has been poor. As well, as a formative tool, the ITER often does not provide trainees with meaningful data on their own strengths and weaknesses. However, in terms of utility as an assessment tool, ITERs have many positive characteristics. They can be reliable, especially if multiple assessors are used to complete the ITER [28]. Qualitative and narrative components on an ITER, if based on specific traits with formative feedback, rather than generalities, can be very useful, and have been shown to be a predictor of overall and long-term competence [28]. Unfortunately, the educational impact on the learner is poor, especially if ITERs are completed long after the fact, without the opportunity for learners to discuss and reflect on areas needing improvement, however, if done on a more frequent basis, with formative feedback, while residents can still act on the results, ITERs have the potential to have a positive educational impact. They are cost effective, and have good acceptability to learners and assessors, as they theoretically are based on actual clinical performance. The challenge, then, is how to improve the ITER as an assessment tool to ensure all of the theoretical positive attributes are achieved.

ITER report quality can be improved with structured feedback to faculty on their ITER completion. Faculty overall feel that ITERs are worthwhile, however, in a study on faculty perspectives on the ITER, Watling et al.[29] found that evaluators felt their ability to produce a meaningful approach to ITERs was compromised by time constraints, lack of continuity between educational assignments, and the challenge of giving negative feedback. These areas need to be addressed in order to improve ratings. Engagement of faculty and residents in the ITER process is a critical factor in ensuring the ratings are improved. Thus, overall ITERs have many of the attributes required for a useful assessment tool, but programs must ensure faculty and residents are engaged in the process in order to get the most useful ratings. They likely will remain an important component of the evaluation system, but perhaps will become a committee-driven evaluation, and may better employ milestones and their assessment in the future.

Mini-CEX

The mini-CEX was developed by the American Board of Internal Medicine as a workplace-based assessment tool that would be feasible to implement in the real clinical setting, be useful for feedback, and give reliable data [30]. In the mini-CEX, the trainee is responsible for selecting a clinical encounter, where they will ask an assessor to observe them in the real patient setting. Thus, a snapshot of the doctor–patient interaction is observed. The assessor collects information on the encounter on a structured evaluation form, with immediate feedback to the resident on their performance. Trainees are responsible for selecting from a range of problem groups and assessors, so over time, they have a collection of assessments in their learning portfolio, leading to more stable (reliable) ratings. History and physical examination skills, communication, professionalism, organization, and efficiency, as well as overall clinical care are covered on the mini-CEX.

An instrument for assessment of surgical skills, with many aspects similar to the mini-CEX, was developed by a group in Ottawa [31]. This tool can be used by surgical programs to help evaluate many of the items important for surgical management.

360-Degree Evaluation

The 360-Degree evaluation is a measurement tool completed by multiple people, with different perspectives, who each interact with the resident. Patients, nurses, allied health personnel, peers, subordinates, and other related specialists might all complete surveys. Generally, these are best used for evaluation of competencies such as interpersonal and communication skills, as well as professionalism. Practical considerations involve the logistics of distributing and collecting the forms (though online programs exist), and ensuring the evaluators are providing reliable data and not simply using this as a “gripe” session. It is important to assess how many of the evaluations being used are needed for the data to be reliable in order to keep the administrative burden acceptable. Usually, 10–20 completed forms are needed for reliable data. Studies in different domains suggest that 360-Degree forms correlate reasonably well with preceptor as well as self-reports of performance [32].

Oral Examinations

The oral examination has traditionally been used to assess clinical judgment. A structured oral examination is likely best to ensure reasonable reliability for formative or summative assessment. Medical knowledge, patient care, interpersonal and communication skills, and to an extent, professionalism can be assessed on an oral examination if it is well structured. As the structured oral remains an important component of board examinations, it is useful to provide residents the experience of participating in this type of assessment. In terms of utility, a structured oral has reasonable reliability and validity, is educational and acceptable to learners and other stakeholders, and is cost effective. Many of these properties will likely ensure it continues to be used for high stakes evaluation for the foreseeable future, thus, it should remain a part of the assessment toolbox for residency programs.

Portfolios

A learning portfolio is an important assessment tool in a competency or outcome-based learning framework [33]. The learner is responsible for developing the portfolio, and providing evidence of learning and achievement related to the competencies that have been mastered. A learning portfolio might include items such as self-evaluations, articles related to specific outcomes, presentations they have made around a topic, and results of both formative and summative assessments such as the mini-CEX, CARSITE examination, or oral examinations, all helping to support promotion decisions. Ideally, this learning portfolio would follow the individual into independent practice, and become part of maintenance of certification.

Technical Skills Assessment

Logbooks and Case Numbers

Logbook numbers can be seen as a surrogate for technical skill assessment. Although increased numbers of cases are associated with improved outcomes, the learning curve varies greatly between trainees, as does the number of cases required for proficiency [34]. Case numbers, other than identifying deficiencies, do not give meaningful feedback to residents on where they need to improve. However, they are useful for training programs to assess the operative experience provided to their residents, and to identify potential deficits.

Procedure-Based Assessment

Procedure-based assessments (PBAs) are completed via direct observations of entire operations [35]. The assessment covers consent, the preoperative planning, preoperative preparation, exposure/closure, intraoperative technique, and postoperative management. The United Kingdom has the most experience with using PBAs, where they are required for many technical specialties. For colorectal surgery, the Operative Competency Committee of the ASCRS has developed PBAs for several technical areas, which are available on the program director’s website.

Because the PBA captures performance in a “real-life” environment, it is an ideal form of evaluation to demonstrate milestone achievement. However, the feasibility of its use as a high stakes assessment is questionable, as residents cannot be left to “fail” the examination. Within a competence-based curriculum, however, PBAs used formatively could help provide evidence of milestone achievement.

Simulation/Virtual Reality in Technical Skill Assessment

Simulation-based assessment can be used to assess many areas of competence, including professionalism [36], team-based skills [37], patient communication and interpersonal skills [38], and technical skills [39], Realistically, the use of these types of simulations will likely be in primary residency programs, where the larger number of residents justify the infrastructure and development costs required.

Simulation of technical procedures can include non-live animal models, synthetic tissue, computer-based models, cadaveric tissue, live animal, and hybrid platforms. An example of a simple model for basic technical skill assessment is the MISTELS system used in the Fundamentals of Laparoscopic Surgery program [40]. Virtual reality simulators have also been used for more complex skills, notably, simulators for colonoscopy and for laparoscopic colectomy. A recent Cochrane review [41] found that VR simulation led to improvements in operative time and “performance,” but the impact on patient outcome was unclear.

GAS and LCAT of the National Training Programme in England

The National Training Programme (NTP) for Laparoscopic Colorectal Surgery in England used the global assessment scale (GAS) for feedback [42] as part of the training program. Once a trainee within the program was deemed proficient, DVD recordings of two independently performed cases were submitted and reviewed by two blinded accredited assessors, using the laparoscopic competency assessment tool (L-CAT) [43]. The L-CAT is comprised of a 16-item marking sheet (4 task components for each of the 4 domains), incorporating 22 items identified. The tool was shown to perform well, with excellent reliability and evidence of validity. However, the examinees were surgeons in practice who performed their cases independently, thus the tool might not be suitable for a residency training program where intervention must occur for poor performance.

Colorectal Objective Structured Assessment of Technical Skills

The COSATS [44] exam is an assessment tool developed for summative assessment of colorectal technical skills. It is a derivation of the OSATS examination [39], which has been used in many specialties. In the COSATS, the candidate performs tasks specific to colorectal surgery, using a combination of virtual reality and bench models to simulate the selected skills.

Performance is assessed at each station using a task-specific checklist and a global rating scale, and pass–fail decisions are made based on the performance across eight stations. To date, reliability and evidence of validity are very good to excellent for a high stakes assessment. However, the COSATS is best seen as a summative assessment, as the feasibility of individual programs developing and setting up this examination for a few residents at a time is questionable.

Global Rating Forms

The global rating scale of the OSATS, and modifications of it, such as the GOALS [45] and the GAGES [46] to assess endoscopic skills, have been used to assess residents’ technical skill in the operating room and in endoscopy, and to assess surgeons in practice [47], all with good evidence of reliability and validity. The global rating form is a relatively straightforward tool to use to collect a series of assessments on the operative skills of residents.

In summary, programs will need a variety of assessments to gather the evidence required by a clinical competence committee to make decisions on milestone attainment by their residents. Resident portfolios will likely become an important component of resident assessment, and shift some of the onus for collecting and collating evidence of competence attainment on the learner. These portfolios could then be carried into practice to aid in the maintenance of certification.

How Can CBME Be Applied to Colorectal Residency Training ?

Colorectal milestones have been developed by the American Board of Colorectal Surgery and the ACGME. Residents are coming into programs with varying degrees of experience and competence, thus it is likely important to assess their starting point (along the milestones early in their residency) to ensure there is time to address areas of deficiency within a short training period. Deficits in technical or other skill sets need to be diagnosed as soon as possible within the training program to ensure adequate progression along the milestones occur.

An example of a Colon and Rectal Surgery Milestone is the Patient Care for Rectal Cancer. Multiple competencies related to rectal cancer care are included in the milestones, including imaging, choice of operating, surgery, and postoperative management and surveillance. For one of the components of competence, an entry level resident would be expected to list some imaging options for TNM staging. This competency would be relatively easy to assess with a quick written or oral examination. As residents progress, they are expected to be able to formulate strategies for imaging the rectal cancer patient and interpret the results. This likely would be taught in the program in several ways, including seminars or readings on imaging of rectal cancer, coupled with multidisciplinary rectal cancer rounds, where imaging is reviewed with radiologists, and other strategies for imaging might be discussed. In contrast, Level 4 (graduating resident) would expect that the resident is able to assess imaging information and justify a TNM-based treatment strategy. Again, this is currently done to an extent in multidisciplinary rectal cancer rounds; however, for a competency-based curriculum, the resident would have to gather data to show that they had been assessed on this competence. A direct observation of the resident’s performance in cancer rounds, an oral examination, or a case write-up in which imaging is used to justify a TNM-based treatment strategy could all be used to assess the resident, and demonstrate competence in this area. The technical components of competence could be assessed with the COSATS global rating scale, or with a PBA for low anterior resection. Multiple components of the patient care competencies would be taught on the oncology rotation, or in the clinical setting, in rounds and during direct patient care. Overall, a move to a competency-based curriculum requires assessment to demonstrate that the resident is mastering the component pieces of rectal cancer patient care, and ensuring they move along the milestone progression expected of them. Assessment in a patient care setting would be expected for many of the components.

As the milestone committee meets, the evidence of competence assessment would be reviewed, and the milestone achievement marked for the resident. If there are competencies that are following below expected, targeted learning interventions towards these areas would be implemented.

For example, if a resident is noted to be deficient in some of the operative components, they could spend time in the simulation center practicing enabling skills for the procedures. The program would then try to ensure they have increased exposure to rectal cancer cases, preferably with known strong clinical teachers, allowing them to use their clinical time for refinement and milestone progression. Medical knowledge deficits identified early might lead to a structured reading program, or use of resources such as CARSEP or CREST.

Residents should develop a learning portfolio, which would include, among other things, direct observation assessments, self and other identified areas for improvement, quality improvement initiatives, and their case log information. Reflection on cases, especially in areas of difficulty, might also be helpful. The learning portfolio allows for the clinical competence committee to have multiple sources of information on the resident, enabling more accurate milestone assessments. For the resident, it allows them to see areas for improvement, enhancing self-reflection, and movement towards excellence.

Conclusions

CBME has arrived and will be an integral part of surgical training. As surgeon educators, we must strive to implement these measures in the most effective way possible and resist the urge to “check the boxes.” Furthermore, we must take responsibility to make trainees understand the importance of both maximizing their learning experience for each case and becoming immersed in the learning environment outside of the operating room. Lastly, we must resist some of our own selfish interests and decide where the education of trainees falls on our priority list. In his essay on leadership [48]. John Maxwell recites, “The Law of the Big Picture” as “People do What People See.” Therefore, it is unlikely for residents to prepare for cases well if they are not receiving instruction during the case or feedback after from disinterested faculty. Trainees must know and perceive that their trainers have a sincere interest in their education, or they will question the sincerity of the feedback and evaluative comments they receive. Although a move to a focus on the outcomes of education in many ways seems daunting, it is educationally sound, and, by moving some of the onus for improvement onto the learner, will hopefully improve lifelong learning in the profession.