Background

The College of Anaesthesiologists of Ireland

The College of Anaesthesiologists of Ireland (CAI) is the training body in Ireland accredited by the Medical Council to deliver the 6-year postgraduate Specialist Anaesthesiology Training (SAT) programme, with an annual intake of 40 trainees. CAI is responsible for developing and refining the national SAT programme, the aim of which is to help develop fully qualified anaesthesiologists who are trained and competent not just in the required technical skills, but in all areas of professionalism. Anaesthesia non-technical skills (ANTS), such as communication or leadership, are integral to clinical competence in anaesthesiology [1].

In the traditional model of specialist anaesthesiology training in Ireland, the time spent in training was considered the most important factor and was assumed to confer proof of competence, with two examinations during the training programme—the membership and fellowship examinations—being the methods of summative assessment. These examinations are designed, however, to focus on and sample primarily the candidates’ knowledge. Progression decisions by CAI are informed by in-training assessments of the anaesthesiology trainees carried out in CAI-accredited hospitals every 6 months. To date, these reports have been prepared by a tutor consultant anaesthesiologist, based on the recommendations of the department of consultant anaesthesiologists within the hospital, rather than on structured feedback in response to direct observation of the trainee’s performance. This model of training is best described as the ‘time spent’ or apprenticeship model, which is similar to many other postgraduate training programmes, nationally and internationally [2].

Multiple changes have occurred in the past few years, however, leading to a shortened training programme with less opportunity for anaesthesiologist trainees to build up their professional experience. Such changes included the transition from a 2-year basic training followed by a 5-year specialist training programme, to a 6-year single entry ‘run-through’ SAT programme in 2014, the implementation of the European Working Time Directive [3], an increased demand for efficiency in operating room time and public concerns about patient safety [2]. These developments have coincided with increasing attention within the specialty to the importance of ANTS and of professionalism [4, 5].

Introducing competency-based Specialist Anaesthesiology Training using EPAs

In this context, the CAI felt it timely to adopt a competency-based approach to postgraduate education for the anaesthesiology trainees. This view is reinforced by the prevailing paradigm shift in medical education and training from an approach focused on structure, process and time served to one focused on competencies [6]. Competency-based medical education (CBME) is an approach to preparing physicians for practice that is fundamentally oriented to graduate outcome abilities and is organised around competencies derived from an analysis of societal and patient needs. It de-emphasises time-based training and promises greater accountability, flexibility and learner-centeredness [7]. Competency-based medical education uses clearly defined learning outcomes, describing the desired abilities of graduates, formulated as competencies [8].

Implementation of CBME, however, has not been without its challenges and critics [9]. In response to many of these, the concept of EPAs has emerged as an approach to achieving the goals of CBME [10]. Following extensive research of international practice and an evaluation of the CAI’s context and needs, EPAs were adopted as a potentially promising framework for implementing competency-based anaesthesiology training in Ireland. EPAs are units of professional work that can be entrusted to the trainee once the trainee has been deemed competent. Trust is a central concept for safe and effective health care [11]. This trust is not only important in the patient-anaesthesiologist relationship but also vital in the trainee-consultant relationship. EPAs are independently executable, observable and measurable in their process and outcome and integrate competencies from across a wide range of domains of professional practice [11]. A key element of any competency framework is a programmatic approach to assessment [12]. EPAs offer a valuable and robust framework for designing assessment based on direct observation or review of performance, with formative feedback [13].

This major change management project not only involved the development of four core anaesthesiology EPAs but also introduced new modes of assessment by adapting four tools of workplace-based assessment (WBAs), with an emphasis on formative feedback. The potential advantage of EPAs for trainees is the opportunity for them to receive feedback on aspects of their performance from their consultants, in a planned and structured way, based on observation and/or review of their practice. The Medical Council of Ireland conducts the ‘Your Training Counts’ survey annually, since 2014, gathering valuable experiences of over 800 trainees regarding their clinical learning environments, across multiple specialities of medicine. The survey found feedback to be the lowest ranking attribute of trainees’ clinical learning environments consistently, year on year, across the specialities, including anaesthesiology. One of the action points from the Medical Council is that feedback to trainees must always be provided and its quality significantly improved.

Professionalism in anaesthesiology

Professionalism is regarded as the cornerstone of professional integrative practice in anaesthesiology and intensive care, with its roots in a robust training and education programme, underpinned by current and innovative research, scrutinised by appropriate tests of knowledge and competencies to ensure the highest standards of patient safety [14].

There is a lack of clear definition, however, of professionalism, leading to challenges in teaching and reinforcing professionalism in practice. Lapses in professionalism are easier to point out, but they too are often vaguely defined [5]. Moreover, the concept of professionalism has been shown to vary from different perspectives [15].

Multiple studies internationally identified hallmarks of professionalism. Hilton et al. believe professionalism to be an acquired state, rather than a trait, and one that takes years to attain and must be maintained throughout a professional lifetime [16]. Hilton et al. identified six domains of medical professionalism, divided into two categories: the personal attributes of professionals (including ethical practice, reflection and self-awareness and responsibility/accountability for actions) and the co-operative attributes of professionals (comprising respect for patients, teamwork and social responsibility) [16].

Surdyk et al. defined professionalism in terms of five overlapping relationships [17]. These include the following: physician-to-patient, physician-to-society, physician-to-health care system, physician-to-physician and physician-to-self relationships. Kearney et al. identified 40 qualities which defined professionalism in anaesthesiology, through a series of three questionnaires [18]. These qualities were grouped under three themes: humanistic (integrity, maintaining confidentiality, respect for patients’ views, empathy), personal development (self-awareness, commitment to lifelong learning, coping with uncertainty, accepting criticism appropriately, maintaining a balance between personal and professional lives) and meta-competences (vigilance, responsiveness, team working, advocacy, communicativeness).

It is clear that, while complex and multifaceted, professionalism should be embedded in our everyday conduct. Role modelling has been the traditional method for fostering professionalism, but there is increasing recognition that the teaching of professionalism needs to be explicit rather than implicit, and not to rely solely on modelled behaviour [19].

The Medical Council Eight Domains of Good Professional Practice

The Medical Council is the regulatory body for doctors in Ireland, whose purpose is to protect the public by promoting and ensuring high standards of professional conduct and professional education, training and competence among doctors. It sets the standards for medical education and training from medical school to postgraduate training and is responsible for the accreditation of postgraduate education and training programmes.

The Medical Council has elaborated Eight Domains of Good Professional Practice as an organising framework for the development of education and training programmes in Ireland [20]. Numerous comparable competency frameworks have been established internationally to guide standards of training [21]. The process of developing such frameworks generally involves empirical research, subject matter expertise and a wide consultative process to ensure the framework reflects societal expectation of the profession. Most notably, the Royal College of Physicians and Surgeons of Canada (RCPSC) has developed the Canadian Medical Education Directives for Specialists (CanMEDS) framework [22], which identifies the abilities physicians require to effectively meet the health needs of the people they serve. These abilities are grouped thematically under seven roles. A competent physician seamlessly integrates the competencies of all seven CanMEDS roles. CanMEDS provides a comprehensive foundation for medical education and practice in Canada, which specialties use when developing competence-based curricula. The generic professional capabilities framework within the UK serves a similar purpose, where the Excellence by Design policy requires that all the professional capabilities—which are described in terms of nine domains—are incorporated into postgraduate curricula [23].

The Medical Council Eight Domains of Good Professional Practice are listed in Table 1 in the Appendix with a description of each domain. This is the extent of detail provided by the Medical Council, enabling training bodies to elaborate these for different specialties. ‘Clinical Skills’ constitutes just one of the eight domains, with the other seven embodying non-technical skills and professionalism. While considerably less detailed and prescriptive than the CanMEDS competency framework, the Medical Council offers an overarching framework comprising ‘domains’ that are akin to the ‘roles’ within CanMEDS. The domains are comprised of generic competencies which have been adopted by specialist training bodies in Ireland as a framework which shapes the development of curricula for their training programmes.

The Medical Council more recently developed a broader categorisation of the principles of professionalism in the form of ‘Three Pillars of Professionalism’ [24], namely partnership, practice and performance (Table 2 in the Appendix). These are values, principles and behaviours the Medical Council expects of all doctors from the moment they enter medical school right through until retirement, so that the highest possible standard of care is provided to patients. The guide sets out the principles of professional practice and conduct that all doctors registered with the Medical Council are expected to follow and adhere to, for the benefit of the patients they care for, themselves and their colleagues.

The Medical Council deploys the Pillars of Professionalism as a schematic to indicate ways in which the application of particular principles and values underpin good care [24]. The Eight Domains of Professional Practice, however, remain the more valuable and commonly used framework in Ireland to inform the design of the curricula for specialist training programmes, including competency-based ones. CAI adopted the Medical Council Eight Domains of Good Professional Practice as an organising framework when developing a new competency-based programme based on Entrustable Professional Activities (EPAs). There has been an increasing focus on the integration of non-technical skills into the anaesthesiology curriculum in Ireland and increasing recognition that non-technical skills are crucial skills that need to be taught, acquired and assessed [5, 25].

Aims

The aims of this major change implementation project undertaken by the CAI were to develop and trial four core anaesthesiology EPAs, to ensure comprehensiveness of each EPA in terms of the range of Domains included, to devise workplace-based assessment tools, (Feedback Reports) for the EPAs and to promote feedback and assessment of the full range of the Medical Council Eight Domains.

In this paper, we outline the process of developing EPAs as the framework for competency-based specialist anaesthesiology training in Ireland, focusing primarily on the tagging of the competencies within an EPA to the Medical Council Eight Domains of Good Professional Practice.

We present the outcome of CAI’s experience of incorporating the Eight Domains into the four trialled EPAs, focusing on the extent to which the workplace-based assessments succeeded in ensuring greater attention to a wide range of domains: clinical skills, non-technical skills and professionalism. Arising from these findings, we draw conclusions and make some recommendations for more effective incorporation of domains of professional practice in the processes of curriculum development, teaching, learning, feedback and assessment.

Methods

Development of EPAs for competency-based specialist anaesthesiology training

The CAI adapted and trialled EPAs as a promising framework for competency-based specialist anaesthesiology training, 2015–2018. An EPA is a unit of professional practice, to be entrusted to a trainee to execute without supervision once the trainee demonstrates sufficient competence [8]. EPAs combine the requisites of competency-based training, safe patient care and autonomous practice. EPAs relate competence to clinical practice and mastering an EPA may be achieved by trainees at different points in time, making EPAs a more flexible and individualised approach to competency-based training.

The CAI adopted a stepwise approach to EPA design, as advocated for curricular design in the literature [6]. A first step in building an EPA-based curriculum is to identify the core EPAs of the profession [8]. The initial scoping of the specialty yielded a provisional list of ten core EPAs and seven advanced EPAs. The decision was made to develop and trial four core EPAs among mostly first-year anaesthesiology trainees (40), namely ‘General anaesthesia for ASA 1 and 2 patients for low-risk surgical procedures’, ‘Vascular access’, ‘Managing pain in labour’ and ‘Paediatric anaesthesia’. These four EPAs were created by EPA development groups of fully qualified anaesthesiologists with expertise in relevant fields and a few advanced trainees, facilitated by a medical educationalist using a process informed by the nominal group method [26].

Each of the EPAs was developed using a template adapted from the work of ten Cate et al. [27]. The EPA template included the following: title, a brief description of the EPA, the pre-requisites to be met prior to the trainee attempting the EPA, the co-requisites to be achieved simultaneously with the EPA and the list of competencies that were to be demonstrated. A unique feature of the CAI template for developing EPAs is that it makes provision for (a) tagging individual competencies within the EPA to the Medical Council Eight Domains of Good Professional Practice, (b) mapping competencies to teaching and learning opportunities within the curriculum and (c) blueprinting competencies against assessment tools. The CAI created a data management system for storage of the EPAs, with functionality for analysis of EPAs, including analysis of the completed EPA Feedback Reports.

The second step involved elucidating all the competencies for an EPA (classified as knowledge, skill or attitude). The CAI adopted the Medical Council Eight Domains of Good Professional Practice as the framework within which to tag the competencies of an EPA. This helped ensure that, with each iteration of the EPA, professional qualities that stakeholders expect of the specialty were included. By reviewing EPAs and the constituent competencies through the lens of these domains, it encouraged a move away from a focus on purely technical and procedural skills and prompted the inclusion and integration of non-technical competencies in a more explicit manner, as essential competencies associated with each professional activity. The outcome was in marked contrast to the existing modular curriculum within CAI, which was organised around clinical themes with ‘professionalism’ as a discrete stand-alone module.

Design of WBAs

The third step in the development process was determining how the attainment of each EPA would be assessed. Competency-based anaesthesiology training programmes require robust assessment of trainee performance and commonly combine different types of WBAs that cover multiple facets of practice [28]. The intended purpose of the WBAs was to monitor and record the developing competence of trainees and to improve the quality of learning, through providing structured feedback.

Four tools—called ‘Feedback Reports’ to signal their intended focus—were developed and adapted from existing best practice in workplace-based assessment. These were the DOPS (direct observation of procedural skills), Mini-CEX (mini clinical evaluation exercise), CBD (case-based discussion) and Fieldnote (Table 3 in the Appendix). The competencies within each EPA were blueprinted to the assessment tool(s) most valid for that purpose. CAI also created a written guide to help select which of the four tools was best suited to assessing the trainee’s performance on a given procedure, case or event. The incorporation of Fieldnotes, adopted from Family Medicine in Canada [29], was of particular value in providing a mechanism for assessing many of the non-technical skills within the EPAs and aspects of professionalism. Fieldnotes were a novel tool in the context of anaesthesia. They provided a tool responding to unplanned, serendipitous events within clinical practice and providing feedback on them. As with the other tools, they could be initiated by either the consultant or trainee, but the focus was on narrative feedback only—level of supervision was not relevant for such observations. The utility of Fieldnotes had been established by an evaluation of an earlier pilot study within CAI, along with the conditions that needed to be in place for effective implementation [30].

Providing structured feedback was an integral part and the main aim of the assessment, through completion of the EPA Feedback Report. When choosing a particular WBA, the consultant indicated aspects of good performance to the trainee, discussed suggested areas for development and stated the level of supervision that the trainee required for that given procedure/event/case. The trainee added their response and an action plan was agreed on with the consultant. Each completed Feedback Report was tagged to the one Domain of Professional Practice to which the feedback most related. This feature was added as a valuable way of tracking the range of domains on which a trainee received feedback, something that would be audited when the trainee’s portfolio of completed Feedback Reports was being reviewed.

An EPA Feedback Report App offered a paper-free technology platform to support every step of the process. When a Feedback Report was initiated on the app, the consultant or trainee selected the relevant EPA and the assessment tool being used. The dynamic connection between the app and the EPA data management system ensured that only the competencies which were blueprinted to that tool appeared on the screen, serving as prompts to the feedback process.

Finally, provisional ‘criteria for entrustment’ were indicated in the EPA, namely, the number and range of Feedback Reports to be completed to the level of independent practice and the minimum volume of practice for elements within each EPA. It was anticipated that these criteria would be reviewed post piloting.

Promoting feedback on and assessment of all domains of professional practice

To support implementation of the four EPAs on a pilot basis, 27 consultant anaesthesiologists across 25 training hospitals in Ireland were appointed and trained in Dublin as EPA tutors in June and September 2017. Their role was to take a lead in the implementation process, supporting fellow consultants and trainees in their hospital. Many of these tutors actively contributed to developing the four EPAs and the design of the Feedback Reports, demonstrating their commitment to and investment in the success of the EPA implementation process.

A cascade model of training was used to help disseminate knowledge about the new EPAs and to support promotion of a culture and practice of providing effective feedback. Practical workshops in giving effective feedback were provided over 2 days, using the advocacy-inquiry method [31]. Special attention was given to the opportunities to provide feedback across a wide range of domains of professional practice. Examples and role-play exercises included, for example, how to provide feedback on a trainee’s performance in the domains of communication, relating to patients and management, including self-management. Video resources were provided on a web-based platform to illustrate these skills and practices. Evaluation of the training provides positive indicators of intended behavioural change on the part of the EPA tutors. A subsequent survey of tutors, consultants and trainees, post-implementation, affirmed this in terms of the trainees’ assessment of the quality and utility of the feedback they received from tutors. Trainees’ assessment of the feedback from other consultants suggests that cascading this behavioural change to others was more challenging.

EPA tutors provided departmental briefing meetings and support to their non-tutor consultant colleagues. These briefings were generally held as part of scheduled departmental meetings, where educational issues were often discussed. Tutors were provided with a pre-recorded video presentation to show at these meetings to ensure consistency of message. This presentation was used to explain the rationale and prompt discussion of proposed implementation of EPA Feedback Reports within the department. The frequency of these meetings was at the discretion of the tutor and was much influenced by the pressures of the service provision of the respective hospital, as well as by the level of the commitment and interest from fellow consultants. As part of this process, it was hoped that a key message from the training—the importance of giving feedback across a wide range of domains—would be imparted. This message was reinforced with regular communication from CAI to all stakeholders, using newsletters which included updates on the range of domains being reported on, encouraging the practice of giving feedback across domains other than clinical skills.

The EPAs and Feedback Reports were trialled, primarily with 40 year 1 trainees during October 2017–May 2018. Completion of eight Feedback Reports was mandatory for year 1 trainees in order to meet requirements for progression. The reports were ‘zero stakes’—the content of the reports did not inform progression decisions in the trial period.

Results

Arising from our analysis of the content of EPAs and Feedback Reports within the EPA data management system, we provide (i) an analysis of the comprehensiveness of the EPAs in terms of domains included and (ii) insights into the range of domains which were the focus of the completed Feedback Reports.

Representation of the Medical Council Eight Domains of Good Professional Practice in the EPAs

The four EPAs comprised between 32 and 44 competencies. Each of these was tagged to one of the eight domains. Following the analysis of the range of domains represented in the EPAs, we found that between 34 and 41% of competencies within the four trialled EPAs were tagged to the ‘Clinical Skills’ domain, followed by Patient Safety and Quality of Patient care (see Table 4 in the Appendix).

Figure 1 in the Appendix demonstrates how 31 competencies within the ‘Managing pain in Labour’ EPA, were tagged, for example. The fact that the ‘Clinical Skills’ domain was the most commonly chosen domain is perhaps unsurprising, as anaesthesiology is a very procedural specialty and there has always been a lot of emphasis on the need to acquire certain basic procedural skills from the start of the training.

Despite all Eight Domains of Good Professional Practice being relevant to our practice, competencies were generally routinely tagged to only five of the eight domains: Patient Safety and Quality of Patient Care, Communication and Interpersonal Skills, Collaboration and Teamwork, Professionalism and Clinical Skills. This concentration is notable as all eight domains are central to the Professional Competence Scheme, whereby all trainees and consultants provide evidence of their professional competence across all eight domains. Our findings suggest that, while it is to be expected that some domains will predominate in specific EPAs, more work is needed to appreciate and recognise the value of each domain and to ensure the description of professional activities integrates all domains of practice.

Analysis of feedback provided in the WBAs

Interrogation of the EPA data management system revealed that in total, 447 Feedback Reports were completed during the 6-month EPA trialling process (2017–2018). We analysed the anonymised data to determine the extent to which the full range of the eight domains was utilised to serve as the focus of feedback (Fig. 2 in the Appendix). Consultants had been asked to indicate the one domain to which the feedback most related. The majority of the Feedback Reports (57%) focused on the ‘Clinical Skills’ domain, while 21.7% were tagged to the ‘patient safety and quality of patient care’ domain. Of note, only four of the 447 Feedback Reports (0.9%) focused on ‘Professionalism’—the domain least utilised as the focus of feedback. Our results identified challenges in promoting feedback across the full range of domains and suggest that defining and assessing professionalism is challenging. Year 1 trainees, however, were the participants in the trialling of the General Anaesthesia EPA—when greatest emphasis is placed on acquisition of procedural skills. This partly accounts for the preponderance of feedback on clinical skills over other domains.

With an emphasis on a steep learning curve for technical skill acquisition in the early years of specialist anaesthesiology training, it is challenging, yet crucial, to maintain a focus on the non-technical competencies that we expect of our fully qualified anaesthesiologists. There is a concern that a focus on skill acquisition, which is easier to describe and to measure, may lead to displacement of important skills such as cognitive and critical thinking, as well as the interpersonal skills needed for effective patient interaction [6].

Conclusions

In the traditional model of anaesthesiology training, formative and summative assessments of trainees focused primarily on knowledge and technical skills. The switch to competency-based training reflects the move beyond knowledge and skills and embraces the broader principles and more holistic approach to what true professionalism entails. The Medical Council Eight Domains of Good Professional Practice is a framework of principles that, incorporated into our curriculum and into the approach to programmatic assessment, can enhance our training scheme to help promote acquisition of non-technical skills and professionalism.

The experience of CAI with tagging competencies within EPAs against Medical Council domains and of naming feedback in terms of the most relevant domain highlights the challenges of interpreting and assessing professionalism in the speciality. Though the Medical Council domains provided a framework that was convenient to incorporate in the EPAs, our experience revealed challenges in utilising the full range when developing the EPAs in the first place and, to a greater extent, when providing feedback. The analysis of the Feedback Reports revealed that tutors and trainees found giving and receiving feedback on what they regard as ‘professionalism’ challenging, as over half of Feedback Reports during the assessment focussed purely on the clinical skills of trainees. This corroborates findings from international literature which suggest that anaesthesiologists find professionalism and communication skills among the most challenging upon which to provide feedback [25, 32,33,34].

Analysis of the data from this change implementation project reveals that though the Medical Council’s set of domains served as a useful framework when developing EPAs and tagging constituent competences, its interpretation and utilisation by anaesthesiologists was limited. The multiplicity of attempts to define professionalism in the literature [5, 15, 17, 35] might suggest the need to define how the principles of professionalism, as elucidated in CAI, can be implemented within the scope of a competency-based curriculum, based on EPAs, with the involvement of key stakeholders. Faculty development offers the opportunity to develop a shared mental model of professionalism within the context of anaesthesiology. The challenge of effecting behavioural change within the trainer-trainee workplace assessment interface is significant in the context of professionalism and non-technical skills. It requires a two-pronged approach: enhancing the skills of trainers in giving effective feedback and strengthening the agency of trainees in seeking and responding to feedback across the full range of domains. Ongoing faculty development and modelling is required, on site and in real time, for consultants who are unlikely or unable to access formal training. Putting the Feedback Report App in the hands of trainees and giving them responsibility for gathering evidence in an e-portfolio of their competence across all domains will be an important element of any future strategy for implementation.

Professionalism is fundamental to medicine’s relationship to society [19]. To help remedy the uncertainty regarding what professionalism entails in our speciality, CAI has devoted a section on professionalism in the new ‘Curriculum for the National Specialist Anaesthesiology Training Programme’. This document explains and puts into context how the Medical Council Eight Domains of Good Professional Practice apply to our profession. A further document put forth by CAI outlined how the Medical Council Pillars of Professionalism translate to anaesthesiology [14]. It is important that we do not return to the situation pre-EPAs, where professionalism was described as a discrete and separate module within the curriculum. Explicit inclusion of professionalism and non-technical competences continues to be a focus for future EPA development. Institutional support and endorsement of same in the form of the active participation of department chairs and CAI is also required to send a message that the subject is important. The definition and description of professionalism must be agreed on and accepted by anaesthesiologists as this provides the cognitive base of professionalism that must be taught. Professional identity arises from a long-term combination of experience and reflection on experience, following years of experiential learning [16], involving reinforcement and internalisation by the trainee of the cognitive base of professionalism.

CAI will continue to provide strong institutional commitment in the form of defining and emphasising professionalism in the formal curriculum and within the EPA-based competency framework. Our role, however, extends beyond this aspect of promoting professionalism. There is an extremely powerful ‘informal curriculum’ that consists of unscripted, unplanned and highly interpersonal forms of teaching and learning that takes place among and between consultant anaesthesiologists and trainees [19, 36]. CAI will continue to avail of every opportunity to reiterate the importance of professionalism in anaesthesiology and to work closely with all stakeholders to promote professionalism. This can be achieved through prescriptive methods such as having professionalism feature explicitly in our written curriculum and methods of its assessment in WBAs, to the more subtle and intangible methods such as encouraging all our anaesthesiologists to reflect on examples of professionalism that they would encounter during their working days. It is our duty to proactively promote and model professionalism in our daily practice as consultant anaesthesiologists and as trainers of the new generation of professionals in anaesthesiology.