Keywords

6.1 Introduction

The traditional model of surgical training was an apprenticeship. The system of apprenticeship first developed in the later Middle Ages and came to be overseen by craft guilds and town governments. A master craftsman employed young people as an inexpensive form of labor in exchange for providing food, lodging, and formal training in the craft. A modification of this historical system was the basis of surgical training for many years and involved a surgical trainee learning initially by observation, followed by a gradual introduction to surgical techniques, initially with careful and close supervision, but latterly with “detached” supervision, perhaps from the theater coffee room. Feedback from the trainer was often intermittent and informal, and the model required and usually achieved extensive operative experience. Apprenticeship based training was therefore suited to a healthcare system where extensive operative experience was available, and in such circumstances, the eventual outcome was usually satisfactory. However, such training was prolonged and often required repeated exposure to a large number of procedures before the trainee became competent to undertake the procedure independently. There was also, inevitably, a potential for the increased risk of complications along the way, especially if the level of supervision was imperfect.

Such a method of learning was never going to be sustainable and there have been a number of drivers for change. The first has been the reduced clinical exposure for surgical trainees that has arisen as a consequence of reductions in working time and increased trainee numbers. A second driver for change has been the increasing need for accountability as a consequence of patient expectations and the requirements of patient safety. A third driver has been a change in educational theory, with the recognition that assessment drives learning [1], combined with an acceptance that the traditional methods had poor validity and reliability. A final driver for change in many countries, but notably in Canada and the United Kingdom, has been regulatory, in that there has been a change in emphasis from traditional time-based curricula to competency-based curricula. As a consequence of this latter, the current surgical curricula in the United Kingdom have 7–8 “indicative” years of training with regular assessments along the way. Theoretically at least, trainees can progress through training at different speeds depending upon their ability, their aptitude, and their exposure.

6.2 The Meaning of Words

A variety of words have been used to describe surgical skill and performance. Words such as aptitude, ability, competency, proficiency, mastery, expertise, and experience are all words that can be used to describe the performance of the surgeon. The difficulty is that many of these words do not have precise definitions and as such, these words sometimes mean different things to different people. For instance, a recent systematic review concluded that there needed to be a clearer definition of what is meant by the term competence when it is applied to surgical performance [2]. For the purposes of this article, the meaning adopted by the UK medical training system will be used, namely that “competence” equates to the minimum skill required to safely and independently practice.

One of the earliest models of skill acquisition was the Dreyfus model. Stuart and Hubert Dreyfus proposed a model that described how learners acquire skills through instruction and training and described five stages of skill acquisition [3, 4]. Although the model was written while they worked within the United States Air Force Office for Scientific Research and is primarily focused upon the development of the ability to fly a plane and even though there have been a variety of academic criticisms, many of the propositions that they made have struck a chord within the surgical community [5]. Using their model, surgical trainees can be described as beginning their training as a “novice” and with learning, supervision, and instruction will progress through the stage of being an “advanced beginner” to becoming “competent.” Within the United Kingdom, surgical training system competency is the lowest acceptable level of performance for certification and independent practice but the Dreyfus model demonstrates that this is not at the end of the line in terms of skill acquisition. With further experience, training and supervision of the higher levels of “proficiency” and “expert” are possible. In some versions of the model a sixth level, “mastery” is included. One way in which this terminology has been expanded to describe the characteristics of a surgical trainee is shown in Table 6.1.

Table 6.1 The principles of the Dreyfus five-stage model of skill acquisition applied to surgical skill acquisition (adapted from [5])
Full size table

A visual image of the progression of a trainee demonstrates the relationship between skill levels and experience (Fig. 6.1). As the trainee gains more experience, then with appropriate feedback, instruction, and learning, their performance levels will improve. Increasingly we are recognizing that not only does surgical experience facilitate skill acquisition, but that simulation can also be used at any point along this curve.

Fig. 6.1
figure 1

A visual model of skill acquisition highlighting the relationship between experience and skill levels [2, 3]

Full size image

6.3 Assessment of Competence

In 1990, George Miller proposed a pyramidal framework for the assessment of clinical competence (Fig. 6.2) [6]. At the lowest level of the pyramid is knowledge (knows) followed by competence (knows how), performance (shows how), and action (does). This model has been the basis for the methodology that is currently used to assess clinical competence. At the lowest level, knowledge is usually assessed by some form of knowledge test such as multiple-choice assessments. Other tests such as simulation tests and Objective Structured Clinical Examinations (OSCEs) target higher levels of the pyramid. The challenge is to devise reliable and valid methods of targeting the upper levels of the pyramid.

Fig. 6.2
figure 2

Miller’s model of performance and its assessment [6]

Full size image

In theory, at least there are a number of ways in which these higher levels of performance of a surgeon can be measured. Firstly, the outcomes of surgical treatment are a potential way of assessing the performance of a surgeon [7]. In practice, however, there are a number of problems with this approach. Firstly, in modern healthcare, the outcome of a patient is typically dependent upon the performance of a team rather than of an individual. Measurement of outcome therefore might not always accurately reflect the performance of the surgeon. Secondly, the existence of comorbidities can enormously affect the outcome for the patient and this variability in case-mix makes comparisons between different surgeons difficult. Finally, the volume of cases that would need to be assessed in order to assess such outcomes is considerable and likely impractical as a means of assessing the trainee surgeon.

Traditional data sources for the assessment of competence can include clinical patient records, administrative databases, and logbooks but all these approaches have their own disadvantages. Review of clinical records is still sometimes undertaken (at least in the United Kingdom), for instance, when the performance of a surgeon is under question by the regulator or employer, but it is time-consuming and expensive and multiple records need to be reviewed for any sensible judgment to be possible. Databases and registries can provide information for the reporting of surgical performance, and the use of such registries has recently been introduced for some surgeons in the United Kingdom to describe summaries of caseload and morbidity. While the early registries were self-completed by the operating surgeon (with all the associated potential for bias) [8] more recent versions have been based around administrative databases and are currently intended to support self-reflection, appraisal, and learning [9]. Finally, surgeons themselves often keep logbooks of their cases, but while they provide excellent measures of volume, they are less useful for the assessment of process and outcome.

In theory, observation of a surgeon at work might be expected to provide the most accurate assessment of their performance, but there is the obvious worry that the presence of an observer might alter the surgeon’s behavior. Accordingly, observation should perhaps either be almost routine or alternatively covert for it to accurately represent the performance of the doctor. It is with this background that the so-called workplace-based assessments (WPBAs) have been developed to assess the performance of a surgeon. For this approach to be effective there are several requirements;

  • The observer should have the clinical expertise to be able to make appropriate judgments. So, for a surgical trainee, it is important that a surgeon is an observer making the assessment of technical competence. In contrast, it could be argued the most important observer of communication skills would be a patient.

  • It is helpful to have both multiple observers and multiple observations when assessing the competence of the trainee since this will increase the reliability of the judgment.

  • It is important that the observer is trained to undertake the assessment appropriately, and additionally to be able to provide appropriate feedback which will facilitate future learning.

6.3.1 Assessment of Competence in Surgeons

In most modern competency-based training systems, WPBAs have become the mainstay of competence assessment. By designing tools that are valid and reliable, a number of aspects of a surgeon’s performance can be assessed. These assessments have a dual purpose; firstly, as a formative tool, to facilitate feedback for the trainee, with good evidence that regular, comprehensive, and well-structured feedback will facilitate learning and enhance the progression of the trainee [1, 10]. However, a second potential role is in the summative assessment of surgical trainees and while individual workplace-based assessments are rarely used in this way, a basket of WPBAs for a trainee, over a period of time, is a good indicator of whether that trainee is progressing appropriately.

There are a number of separate components of a surgeon’s performance that can be assessed. Firstly, and most obviously there is the technical competence of the surgeon but given that most surgeons spend only a proportion of their time in the operating room it is also important to assess clinical competence in their interactions with patients in other settings. It has also become clear that non-technical skills such as decision-making, leadership, teamwork, and communication skills also affect the performance of the surgeon and in recent years assessment of these non-technical skills has moved forward considerably. Any new WPBA must undergo a formal evaluation to confirm its feasibility, acceptability, validity, and reliability. To ensure face validity they should comprise direct observation of workplace tasks while for reliability to be confirmed there should be multiple measures of outcomes using several observers with frequent observations. Any assessment needs to be feasible within the context of the training and working environment and the intention was that once the trainers had been trained in the use of the assessment process, they would be cost effective.

There are a variety of WPBAs that are routinely used in different countries and in different specialties globally. In order to try to demonstrate how they can be linked together to deliver a rounded, holistic assessment of the performance of a surgical trainee, the system used to assess surgical trainees as they progress toward certification in the United Kingdom is described below. There have been many variations of this model described but the principles underlying each system are largely similar.

6.3.2 Competence Assessment in Surgical Training in the United Kingdom

A competency-based curriculum was introduced in the United Kingdom in 2007, providing a framework for surgical training through to consultant level. There was a syllabus that defined the knowledge, clinical judgment, technical and operative skills and professional skills and behaviors that were needed in order to progress. The curriculum was accessible online [11] and contained the most up-to-date versions of the specialty syllabuses. Some aspects of the early years’ syllabus were common to all specialties, but were increasingly singular as training in each discipline advanced. The curriculum was founded on a number of key principles including

  • A common format and similar framework across all the specialties,

  • Systematic progression through to the certification,

  • Standards that were underpinned by robust assessment, and

  • Regulation of progression through training by the achievement of outcomes that were competence-based rather than time-based.

The purpose of the assessment system was first to determine whether trainees were meeting the standards of competence and performance specified at various stages in the curriculum, secondly to provide comprehensive feedback to the trainee, and thirdly to determine whether trainees had acquired the knowledge, clinical judgment, technical skills, and behavioral and leadership skills required to practice independently. The individual components of the assessment system were WPBAs covering knowledge, clinical judgment, technical skills and professional behavior and attitude (Table 6.2), a surgical logbook, knowledge-based examinations, learning agreements, and the supervisors’ report with a summary annual review of competence progression. In recent years additional workplace assessments have been added including assessment of teaching and an assessment of audit.

Table 6.2 Workplace-based assessments used in the UK surgical training system
Full size table

The WPBAs were criterion-based with the primary purpose being to provide feedback between trainers and their trainees [1, 10]. They were designed to be trainee-driven but inevitably there were occasions when they were trainer-triggered. The accumulation of WPBA outcomes was one of a range of indicators that informed the annual review. As a consequence, a decision could be made whether there had been satisfactory progression and consequently whether the trainee could progress or complete training. The trainee’s educational supervisor had a key role in judging whether the trainee required more than the minimum number of assessments. In principle, the assessments needed to be started early and continue regularly with the expectation that there would be evidence of progression throughout the training period. All the assessments in the curriculum included a feedback element. Educational supervisors were able to provide further feedback to each of their trainees through the regular planned educational reviews and appraisals that occurred at the beginning, middle, and end of each placement, using information contained in the trainee portfolio and feedback from other trainers in the workplace.

6.3.3 Assessment of Technical Skills

For surgeons, it is perhaps inevitable there has been a historical focus on the assessment of technical skills. The most widely used WPBA in this context is probably the objective structured assessment of technical skill (OSATS) which was developed to assess the performance of Canadian surgical trainees and includes seven operative competence scores; respect of tissue, time and motion, instrument handling, suture handling, the flow of operation, knowledge of procedure operative performance, and final outcome [14]. There are now many variations on the OSATS scale including the operative performance rating scale (OPRS) [15] and the global rating index for technical skills (GRITS) [16].

The procedure-based assessment (PBA) was originally developed by the Orthopaedic Competence Assessment Project in the United Kingdom [17] and has since been adapted for all surgical specialties [12]. The assessment method uses two principal components: a series of competencies within five domains and a global assessment that was initially divided into four levels but has now been expanded somewhat to include assistance at an operation (Tables 6.3 and 6.4). In contrast to many other technical skills tools, there are domains for preoperative planning (including consent) and post-operative planning. The highest rating within the global assessment is the ability to perform the procedure to the standard expected of a specialist in independent consultant practice within the UK National Health Service.

Table 6.3 The domains of the Procedure-Based Assessment [12, 17]
Full size table
Table 6.4 Global assessment of the PBA [12, 17]
Full size table

Whichever tool is used there is value in obtaining multiple assessments from multiple observers. For instance, the initial validation study of the PBA suggested that there was excellent reliability when more than three assessments were used for a particular procedure or when two observers each undertook two assessments [12]. Because the PBA is procedure-specific, all of the core surgical procedures within a specialty-training pathway need to be assessed separately.

There remains interest in other, more automated ways of measuring operative competence [18, 19]. For instance, it is possible to analyze a surgeon’s movements in a variety of ways including the use of sensors attached to the surgeon’s hands and this approach has been used on the da Vinci robotic system. This sort of approach has suggested that experts use fewer, smoother movements and that they manipulate tissues more gently.

6.3.4 Assessment of Non-technical Skills

In recent years, there has been increasing emphasis upon the ability to measure the non-technical skills of a surgeon. We know that there is good evidence that when analyzing adverse events in healthcare, we see that many of the underlying causes reflect non-technical aspects of performance rather than a lack of technical expertise. These non-technical skills might be defined as “those critical cognitive and interpersonal skills that underpin technical proficiency.” The most widely used tool in the theater environment is the non-technical skills for surgeons (NOTSS) instrument, which has four domains: situation awareness, decision-making, communication and teamwork, and finally leadership (Table 6.5) [20, 21] (more details in Chap. 17). The NOTSS tool can be used by the surgical supervisor but there is often added value from using other members of the theater team to additionally assess the trainee.

Table 6.5 NOTSS summary rating form [20, 21]
Full size table

6.4 Challenges and Future Directions

The introduction of competency-based training in the UK exemplifies some of the challenges that can occur [22, 23]. First, it is essential that the training faculty be trained to use the tools appropriately. If the trainers do not know how to use the assessment tools properly, then the results of those assessments will be inaccurate. In the United Kingdom, following the “big bang” introduction of competency-based training in 2007, it was some years before many consultant trainers were trained to use the WPBAs, although this now has been achieved. A second problem has been the tendency by trainers and trainees alike to view these tools as a “tick-box” exercise, with inadequate emphasis upon delivery of formative feedback and with the consequence that the intended learning for the trainee is not achieved. Thirdly there has been a (perhaps) natural reticence for trainees to avoid receiving negative feedback. As a consequence, there has been a tendency for trainees to leave their assessments until they feel that they have mastered the technique, thereby ensuring a positive outcome to the assessment. At the same time trainers, not always wishing or comfortable in providing negative feedback, might not always identify areas for improvement by the trainee. As we move forward, there are still quite a variety of views on when and how frequently assessments should be undertaken [2] and we do perhaps need to understand these issues better.

6.4.1 Entrustable Professional Activities

Another area of difficulty reflects the granular nature of the WPBAs. They were designed to assess relatively small components of the daily activities of a surgical trainee. The difficulty comes in trying to translate these assessments into day-to-day clinical practice. One concept that has sought to resolve this problem is the concept of the entrustable professional activity (EPA) [24, 25]. All (certified) clinicians make daily judgments regarding the trainees with whom they work and what they “trust” them to do on their own and to what extent they require supervision. The EPA uses this principle to describe the extent that a supervising surgeon will trust the trainee to undertake a piece of work. A definition of an EPA might be “a unit of professional practice that can be fully entrusted to a trainee, once he or she has demonstrated the necessary competence to execute this activity unsupervised.”

As such the intention is that EPAs are not intended to replace WPBAs, but instead to translate them into clinical practice by describing different types of work. So, for example, while a WPBA assesses whether a trainee is competent to take a history from the patient with a particular clinical problem (i.e., it is a descriptor of the physician), the EPA judgment is whether the trainer trusts the trainee to undertake an outpatient clinic independently (i.e., it is a descriptor of work). Such a judgment will inevitably involve assessment of the trainee’s knowledge, of their interpersonal skills, of their professionalism, and of their clinical skills, all of which might have been previously assessed by a basket of WPBAs.

6.4.2 The Role of Assessment in Simulation

There is good and increasing evidence that simulation, both technical and non-technical, can enhance learning and aid progression [26]. There is a natural tendency to believe, for instance, in relation to technical skills, that simulation has its primary role in the early part of surgical skills training but there is increasing evidence that appropriate simulation can be helpful in all stages of the transition from novice to competent to proficient to expert. However, for simulation to have the maximum effect the same principles of assessment should apply. Assessment will, after all, drive learning and therefore appropriate assessment with appropriate feedback during a simulation exercise will enhance progression. Many of the tools described above, such as the PBA, can be used in a simulated setting but a number of additional tools (so-called simulation-based assessments or SBAs) have been developed specifically for the simulated environment [26]. Such tools should ideally predict real-world performance, although at present that has not conclusively been demonstrated. A systematic review of the association between simulation and patient outcomes concluded that while there was often a correlation between the two, if there was a marked variation in trainee performance, then that translated into weaker performance [27].

6.5 Summary

Although historically, surgical training was delivered via an apprenticeship model, multiple drivers have now dictated that surgeons now need to demonstrate their competence in order to be certified to practice independently. There are a number of feasible, acceptable, valid, and reliable tools that have been developed to assess the clinical, technical, and non-technical competence of a surgeon and these are now widely used in training programs around the world. Although there remain some problems with the implementation of competency-based programs they remain the likely future direction of assessment within surgical training. In the near future the concept of “entrustable professional activities” will likely be used to translate these competencies into clinical practice.

Key Points

  • It is generally accepted that for a surgeon to practice independently, he or she will require a range of clinical, technical, and non-technical skills.

  • There are a range of validated tools, called workplace-based assessments, to assess clinical, technical, and non-technical skills.

  • Many training programs have introduced workplace-based assessments as a central component of competency assessment.