Since its inception, continuing professional development (CPD) for the surgeon has had at its core a paradox: the standard approaches for surgeons to learn and adopt new techniques and procedures have been anything but standard. Hands-on (HO) courses at national meetings have been characterized by a lack of instructional uniformity, an absence of learner-centered educational goals, and inconsistent long-term follow-up. Not surprisingly, this type of approach to surgical skills acquisition has led to disappointing post-course outcomes [1, 2]. From its founding in 1981, the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) has emphasized training outside the operating room prior to performing laparoscopic procedures [3]. As a result, SAGES has striven to create opportunities for its member surgeons to participate in skills-based courses at the SAGES annual meeting. The rates of technologic advancement and procedural innovation have outpaced the ability of surgeons to learn new techniques and procedures using traditional skills acquisition models.

At the 2015 SAGES annual meeting in Nashville, TN, a new approach to HO procedural courses was introduced. The acquisition of data for outcomes and procedure transfer (ADOPT) model was implemented for a portion of participants taking the HO Hernia course in an attempt to improve the limited rates of transfer to practice after standard HO courses. The pilot ADOPT program attempted to enhance procedural adoption rates as well as surgeon competence and confidence through a systematic approach that included longitudinal mentorship, standardized instructor training, and learner-focused education. Pilot participants reported significant improvements in confidence and procedural adoption compared to peers who had the standard 1-day training at the SAGES annual meeting [4]. Additionally, ADOPT participants completed more learned procedures over the next 3 months than their peers in the standard course group (median 26 vs. 7, p = 0.054). Finally, ADOPT participants reported an increase in both confidence and satisfaction with the program, and they felt the course useful [4].

Based on these positive results, the ADOPT pilot program was expanded to include all participants enrolled in the 2016 SAGES annual meeting HO Hernia course. Utilizing the same methods for instructor training and longitudinal mentoring for all course participants, the effectiveness of the program was evaluated by determining procedure adoption rates, participant confidence, and program satisfaction. As with the 2015 pilot, we expected that participants who took the HO Hernia course using the ADOPT methodology in 2016 would report performing more related surgical cases and gaining improved confidence in learned procedures after completing the program.

Materials and methods

Course implementation

The ADOPT HO Hernia program began at the SAGES annual meeting March 17, 2016 in Boston, Massachusetts. This program structure was similar to the 2015 ADOPT pilot program [4]. Briefly, following collection of pre-course benchmark data, all participants in the HO Hernia course underwent structured, learner-focused instruction during the cadaveric lab using the Set-Dialogue-Closure model of the United Kingdom (UK) Laparoscopic colectomy Train the Trainer (Lapco TT) program [4,5,6]. Following this onsite course, participants were enrolled in a year-long program involving longitudinal mentorship, webinars, conference calls, the creation of an interactive Facebook page, and coaching (Fig. 1).

Fig. 1
figure 1

Timeline of the 2016–2017 acquisition of data for outcomes and procedure transfer (ADOPT) hands-on hernia course

Full size image

The faculty were originally recruited based on their expertise in the field, educational background and ability, as well as interest and willingness to participate in the ADOPT program. To be eligible to serve as instructors, all faculty completed a standardized “intra-operative” teaching course structured on the Lapco TT format. For the 2016 program, most of these faculty members had served as instructors for the 2015 pilot program and had already received TT training [6]. The remaining faculty had received the Lapco TT training separately. All received a 2 hour refresher course in the Lapco TT teaching methodology the day prior to the 2016 HO Hernia course.

Ethical considerations

This assessment study of the ADOPT program effectiveness was designated by the Inova Health System Institutional Review Board (IRB) as exempt under category 1: educational practices.

Data acquisition

Program effectiveness was evaluated using the model developed by Kirkpatrick [7]. Level 1, participation, was assessed by course attendance, survey responsiveness, and reported involvement in longitudinal mentorship and webinar activities. Level 2, change of attitudes, knowledge and skills, was evaluated via pre- and post-intervention surveys given to participants 2 weeks prior to the HO Hernia course as well as 3 and 9 months after the course. Information obtained included participant demographics, training, experience, self-reported case volumes, and confidence in procedures taught during the HO Hernia course. Confidence levels were assessed using a 5-point Likert-type scale: 1 = not confident at all to 5 = completely confident. Level 3, behavior change, and Level 4a, changes in practice, were assessed by calculating annual totals of targeted procedures based on the participants self-reported case volumes 3 months prior to and 9 months after the HO Hernia course. In addition, as part of the 3-month follow-up survey, participants reported barriers to implementation and adoption of procedures.

Participant reactions were also evaluated using qualitative data in open-ended responses on the 3-month survey and via participant reflections on the ADOPT program at the conclusion of the mentorship program year. Reflections included essays written in response to an inquiry on the program’s impact on each participant’s practice. In this manner, more nuanced influences of the program could be gleaned as part of the post-intervention survey.

Data analysis

Through descriptive and analytic means, we evaluated participants’ reactions to the course, changes in confidence, and adoption of learned procedures. For reported procedure counts, volumes were annualized prior to comparative analysis. For questionnaire responses, descriptive statistics, including means, ranges, counts, and proportions were calculated. A comparison of annual procedure volume estimates and participant confidence levels prior to the course vs. 9 months after the course was then performed using multilevel mixed-effects linear regression with a random intercept for participants’ repeated measures. We evaluated non-response bias by comparing characteristics of respondents and non-respondents using non-parametric Wilcoxon rank sum tests. Statistical significance was assessed at the level of p < 0.05. Stata/MP 14.2 (StataCorp, College Station, TX) was the software used for the quantitative analysis.

For qualitative analysis, open-ended responses were transcribed then evaluated using the methods of Miles and Huberman [8]. Responses were first read, then re-read, listed, coded, and analyzed for themes. Trustworthiness was achieved through data triangulation, use of participant quotes, and looking for discrepant cases.

Results

Participants

Twenty surgeons participated in the SAGES ADOPT 2016 HO Hernia Program. Of these 20 participants, 17 completed questionnaires prior to the course (response rate 85%); 10 responded to the 3-month follow-up questionnaire (50% response rate) which was completed between 96 and 134 days after the course, and 4 completed the 9-month follow-up survey administered in January 2017 (20% response rate). Table 1 lists participant characteristics. In brief, participants had been in practice on average 15 years; most served at a community hospital in a large urban setting and had an academic appointment. Participants who responded to the 9-month survey had significantly higher pre-course confidence levels compared to non-responders in two procedures: performing open, transversus abdominis release (1.93 vs. 3.0, p = 0.014) and employing components separation via an open approach (2.57 vs. 3.5, p = 0.052), suggesting evidence of non-response bias. Prior to the course, participants reported low counts of components separation techniques (Table 2). Annualized case numbers for component separation were significantly lower than primary ventral hernia repair, recurrent ventral hernia repair, mesh insertion for ventral hernia repair, umbilical hernia repair, and inguinal hernia repair (all p < 0.007).

Table 1 Course participants
Full size table
Table 2 Annual procedure volume prior to the hands-on hernia course
Full size table

Participation and reactions

Participants reported positive reactions to the topics and techniques covered in the course. Both open components separation and transversus abdominis release were most frequently cited as useful learned techniques in participants’ reflections. Participants reported that HO training for open techniques was helpful. Participants also benefited from mentors’ direct guidance as well as discussion of postoperative complications.

Qualitative analysis revealed the importance of mentorship that the longitudinal aspect of the ADOPT Program provided. For example, one surgeon wrote:

It was a wonderful experience to know that I had a forum where I could ask clinical questions…. I feel that the mentorship provided by the ADOPT program has helped me tremendously through my first years of being an attending. I now have more confidence for future hernia repairs.

Additionally, the theme of the utility of being part of a community of learners was apparent. Webinars provided a supportive environment in which to give advice and receive feedback. For example, after presenting a complex case of a patient with a complication from hernia surgery and multiple comorbidities during one webinar, one surgeon noted that “the group seemed supportive and gave me tips and things to watch out for during the future second stage of the operation.”

The degree of mentorship varied. At 3-month follow-up, half of respondents had participated in mentorship calls (n = 2, 20%) or webinars (n = 3, 30%). Among those participating in such activities, 4 (80%) reported having contact with their mentors between one and five times within 3–4 months following the hands-on course via phone, text, email, webinar, or the group Facebook page established for the program. Some participants noted that busy schedules were difficult to overcome in order to find mentorship time.

Among suggestions to improve the program, two respondents (20%) suggested more focus on posterior component separation. Other participants indicated a desire for more information on approaches to recurrent inguinal hernia repairs, laparoscopic component separation techniques, and discussion of decision making between repair options.

Attitudes, knowledge, and skills

Prior to the course, participants reported the lowest confidence in employing a components separation technique with a minimally invasive approach (Fig. 2). Following the course, participant confidence significantly increased for each major objective of the HO Hernia course (p ≤ 0.019). Quotes from the participants indicated the specific nature of their enhanced confidence:

Fig. 2
figure 2

Participant confidence improvement following the hands-on hernia course

Full size image

I am more willing to take on hernia cases which require component separation.

[The] cadaver lab… was a great experience. I tried out a [transversus abdominis release] procedure for the first time on the cadaver and had the opportunity to interact with other surgeons and my mentor.

I have taught my colleagues about the ADOPT 1-year course. I plan to increase the number of ventral hernia surgeries this autumn.

Behavioral and practice changes

Nine participants (90%) reported changes in their practice 3 months after completion of the course. Specifically, four respondents (40%) described performing more procedures, including transversus abdominis muscle release, two (20%) reported performing higher quality procedures (citing “better techniques”, and “better outcomes”), and two (20%) conveyed confidence improvements.

In the extended follow-up survey at 9 months, significant increases in the annualized procedural volumes were reported for open primary ventral hernia repair, open components separation technique, and mesh insertion for ventral hernia repair (p < 0.001, Fig. 3). Annualized reported counts for other procedures did not significantly increase following the course.

Fig. 3
figure 3

Annual procedure adoption increased following the hands-on hernia course

Full size image

Barriers to implementation

Three months after the course most participants, 6/10 (60%), reported no barriers to practicing tasks and adopting procedures learned in the HO Hernia course. Two participants (20%) reported that low procedure volume or low priority of procedures were barriers to adoption. One other participant reported difficulty videotaping cases, and lacking support from senior surgeons and administrators.

Discussion

Evaluation of the ADOPT program applied to an entire HO Hernia course at the SAGES annual meeting demonstrates its effectiveness on multiple Kirkpatrick levels. Learners found the program useful and beneficial (Level 1 effectiveness), demonstrated significant increases in confidence performing taught tasks and procedures (Level 2 effectiveness), and were able to significantly increase their annually adjusted case volume for key hernia-related procedures: open components separation, open primary ventral hernia repair, and mesh insertion for ventral hernia repair (Level 3 effectiveness).

A key component to the success of the ADOPT program appears to be the mentorship between learners and faculty. In fact, the importance of mentorship in surgical education is widely recognized. In their systematic review of mentor–mentee relationships in surgery, Entezami et al. [9] found, consistent with other authors, [10,11,12] that the most commonly discussed qualities of a good mentor were acting as a professional role model; being compassionate, kind, and supportive; acting as a critic, evaluator, and assessor; being a leader in the field; and challenging the mentee. Entezami et al. also described several barriers to effective surgical mentorship relationships: (1) time constraints; (2) scarcity of qualified mentors; (3) lack of mentorship training; and (4) differences related to gender, culture, or generation [9]. The ADOPT program fostered such good mentorship in two important ways: (1) the promotion of quality mentors who were recognized as leaders in the field of hernia surgery, and (2) the identification and mitigation of potential barriers to mentorship (e.g., standardized instruction of all faculty in how to deliver effective feedback and advice and selection of faculty from a diverse background of SAGES’ surgical leadership).

The expansion of the ADOPT program to a full HO course was undertaken based on the belief that longitudinal mentorship and structured standardized teaching in HO courses improve both post-course procedure adoption rates and surgeon confidence levels, leading to better access for patients to surgeons offering the most effective procedures with the best outcomes. Our first ADOPT pilot program in 2015–2016 supported this philosophy [4]. Similar to the pilot program, the expanded 2016–2017 ADOPT program resulted in both increased case volumes and increased confidence levels for performing procedures taught post-course as compared to pre-course. Specifically, of the five procedures for which participants were asked to give volume estimates, over 100 cases were reported at the post-course survey, with the greatest increases seen in laparoscopic primary ventral hernia repair and mesh insertion for ventral hernia repair. Clearly, these findings reveal how completion of a structured, longitudinal program involving mentorship can improve transfer of learned procedural skills to actual clinical practice.

The improvements in both case numbers and confidence levels are in large part due to the engagement of ongoing mentorship. Compared to historic controls from previous non-ADOPT hands-on courses, where no mentorship program existed, [1, 2] a fundamental change is apparent in the educational and practical outcome of the course. The results seen here, consistent with our previously reported results from the pilot program [4] demonstrate not just the benefit, but the necessity, of longitudinal mentorship. As the rate and complexity of new surgical procedures continue to rise, the resource allocation of both the providing organization and the participating surgeon will demand an effective method of procedure adoption. While the use and training of mentors require increased resources, if the goal of the course is to improve the confidence of the surgeon, increase the likelihood of procedure adoption, and improve the availability of these procedures to the patient, then the return on investment for conducting training courses along the ADOPT methodology is clear.

Another important component in the success of the ADOPT program in promoting transfer of skills to clinical practice relates to the community of practice that it helped to create. Lave and Wenger have described the concept of situated learning in which learning occurs as a result of participation within the activities of one’s community [13]. Learning occurs through engagement with other members in one’s specific work environment, in which specific people share common goals; thus, the social interaction itself serves as a vehicle for learning [14]. Such communities of individuals sharing common goals enhance the learning of the participants [15, 16]. For the ADOPT program, a community of practice was in essence created through the numerous activities occurring as part of the longitudinal mentorship (i.e., webinars, talks, mentee–mentor interactions, and interactive Facebook page discussions). In this manner, participants felt comfortable discussing barriers and challenges in order to come up with solutions.

In addition to identifying elements leading to the success of the ADOPT program, barriers to successful implementation were also examined. Fortunately, they were not universal, as a majority of participants (60%) reported no barriers to implementation in the 3-month follow-up survey. Of the remaining responders who mentioned barriers impeding transfer to clinical practice, two (20%) mentioned low surgical volume for learned procedures; the other two (20%) stated that administrative and technical support for implementation was lacking. Pre-registration communication between the participant and their hospital administration prior to enrollment in order to align the goals of both parties related to procedure adoption could help to overcome this latter barrier.

Limitations related to our program evaluation do exist. Similar to the pilot ADOPT program, the numbers for this group of learners remained small, in part due to purposefully limited enrollment in the course as well as partial survey response rates. Low response rates may bias results; evidence of non-response bias suggests that less confident participants did not complete long-term follow-up. It cannot be entirely assumed, however, that non-responders did not encounter similar results; those with low confidence had more potential for improvement. Even so, the effectiveness of the program is still apparent. Some self-selection bias will always exist in a course in which the participant chooses to enroll.

In conclusion, our analysis demonstrates that the expansion of the ADOPT program to an entire HO Hernia course is both feasible and beneficial, with evidence of Kirkpatrick Levels 1–4a training effectiveness. Most encouraging is the evidence of behavioral change in participants’ practice patterns demonstrated by increased adoption of specific hernia procedures. This finding, combined with the increase in confidence that participants demonstrated (Level 2 effectiveness), underscores the importance of standardized instruction by trained faculty, longitudinal mentorship, and the creation of a community of practice/learners as a forum for discussion and learning. The ADOPT program’s expanded success suggests that it is a useful blueprint for the CPD of surgeons wishing to learn new techniques and procedures for their patients.