Assessing surgical technical skills in a structured manner is a topical issue in light of revalidation, shorter and higher surgical training periods, and recent high-profile medicolegal cases. The General Medical Council [15] is implementing compulsory revalidation for every doctor every 5 years. This is mainly a paper process. However, it does not evaluate surgeons’ operative technique or skill. The ability to assess a surgeon’s technical skill in a structured manner would be beneficial for this process.

Technical skills self-appraisal previously used video tapes of operations by trainees/residents and consultants/attending [3, 17], with the surgeon evaluating his or her own technical skill performance using structured assessment criteria. However, to date, no analysis has assessed the detailed self-appraisal of expert surgeons.

A prospective, objective method for analyzing surgical technical skills error can take the form of hierarchical task analysis (HTA). This technique, with its origins in industry, combines task analysis with objective and systematic assessment of errors to permit analysis of procedures that previously were prohibitively complex [9, 11, 20]. Hierarchical task analysis allows a systematic breakdown of the complex surgical procedure for a clear definition of the actions used, allowing easier elicitation of associations between specific actions and errors and permitting more directed evaluation of technical skill [6].

In recent years, HTA has become increasingly popular in studies evaluating surgical techniques. Previous studies concentrated on analyzing surgical procedures, using task analysis to understand the complexities of procedures, with an aim to improve training systems [4, 5, 13, 16]. Studies combining task analysis of procedures with error analysis have focused on surgical trainees, and thus have detected the skill-based errors that constitute the greater proportion of novice errors, implying targets for improved teaching of skills [10, 12].

Some studies have analyzed technical errors in laparoscopic surgery, using task analysis and technical skill scales [8, 18, 21]. However, applying HTA to laparoscopic operations performed by expert surgeons and using it to self-appraise each individual surgeon has not been done to date. The current study evaluated the practicality, feasibility, and reliability of using HTA in the self-appraisal of surgeons performing complex laparoscopic operations.

Methods

Because laparoscopic cholecystectomy is the most common laparoscopic procedure in the United Kingdom and worldwide, it was the chosen laparoscopic operation. The consultant/attending surgeons recruited all had performed more than 150 unsupervised laparoscopic cholecystectomies. The surgeons were recruited from the Imperial College network of hospitals in West London.

A template hierarchical task analysis of laparoscopic cholecystectomy (Table 1) was constructed by using textbooks, articles, papers, Web pages, surgical skill course manuals, and expert panel discussions. The HTA also was evaluated and modified if required by each of the surgeons participating to assess whether it differed from their own prescribed set of tasks for completion of the operation. The task analysis was performed to a level that described the tasks and subtasks required to achieve the goal, but did not describe the technique and instruments that should be used. This was done so that the natural style of the surgeons was unbiased by any implied technique.

Table 1 Standard surgical hierarchical task analysis of laparoscopic cholecystectomy
Full size table

Full-length versions of operations were recorded on digital videotape. These were then transferred without editing onto DVD using Sony Click to DVD Software (Sony, Tokyo, Japan). Two observers assessed each operation blindly and independently. These observers had formal training by an expert in hierarchical task analysis. Both observers watched a pilot batch comprising five DVDs of laparoscopic cholecystectomies together so they could agree on criteria. They watched the DVDs on large-screen televisions and assessed them using the consultants’ modified standard HTA of the laparoscopic cholecystectomy. All the operations were seen, and only slow (nonessential) parts of the operations were fast forwarded.

The expert surgeons had no time limit for completing the laparoscopic cholecystectomy. To be included in the study, each expert surgeon had to complete each major task satisfactorily using the template HTA as a guide. The HTAs of the operations were not graded, but were used to develop an assessment tool for grading technical skills in laparoscopic surgery [19]. This assessment tool is an amalgamation of the template task analysis and the 10 individual expert surgeons’ individual HTAs. This was done to avoid bias by one expert surgeon in constructing the assessment tool so the tool could be potentially used on a large scale, and 7to determine a level of technical skill to maintain surgical certification.

Statistical analysis

Data were collated in an Excel database (Microsoft, Redmond, WA, USA). Statistical analysis was performed using the SPSS software statistical package (SPSS, Chicago, IL, USA). Kappa coefficients were used, and r values greater than 0.61 were deemed indicative of significant reliability. A p value less than 0.05 was regarded as statistically significant.

Results

There were no major postoperative complications. No laparoscopic cholecystectomies were converted to open surgery, and two patients had on-table cholangiograms. A total of 40 laparoscopic cholecystectomies were assessed, involving 14 men and 26 women with a mean age of 56 years (range, 23–69 years) and mean body mass index (BMI) of 27. The American Society of Anesthesiology (ASA) classifications of the patients ranged from 1 to 3. Gallbladders were graded as follows: grade 1 (thin walled with no adhesions), grade 2 (thin walled with adhesions), grade 3 (thick walled), grade 4 (thick walled and chronically inflamed), and grade 5 (thick walled and acutely inflamed). There were 40 cases classified as grades 1 to 3, and none as grades 4 to 5.

There were 10 consultant/attending surgeons recruited to the study, including 9 male surgeons and 1 female surgeon, 9 right-handed surgeons and 1 left-handed surgeon. All the operations were performed unsupervised by the consultant surgeons. There were 40 consultant episodes (mean, 3; range, 6–2).

The interrater reliability for the two observers had a k value of 0.84 (p < 0.05), and the mean intrarater reliability between the surgeons and the observers had a k value of 0.79 (range, 0.71–0.86; p < 0.05). The mean time required by the surgeons to complete the operation was 32 min (range, 15–70 min).

Discussion

Over the past few years, attempts have been made to create technical skill assessment tools for surgery [14]. Initially, these were on-bench models and virtual reality simulators [1]. They subsequently were transferred to real live operations [7]. However, these tools focused mainly on trainee surgeons, assessing their generic technical skills.

It has been shown that expert surgeons can make technical errors in real live laparoscopic operations [18]. Therefore, a practical tool for self-appraisal of technical skills may be of practical use, especially with the General Medical Council implementing the compulsory introduction of revalidation and appraisal of doctors in the United Kingdom.

Appraisal and self-appraisal for the assessment of technical skills and errors in surgery have been topical, with a few studies highlighting their value [3, 17]. Recently, there has been an attempt to assess consultant surgeons on the General Medical Council’s Performance Procedures, but these have involved only on-bench models [2].

In the current study, we assessed the practicality of using HTA as a self-appraisal tool in laparoscopic surgery. The current study demonstrates that the HTA of individual surgeons has face and content validities. It seems to have good interrater and intrarater reliability. Although it is not practical to use this process of self-appraisal for every operation performed by consultant surgeons, it could easily be performed at sequential intervals throughout the surgeon’s working year. An HTA could be constructed for the portfolio of operations each surgeon performs regularly in his or her surgical practice. This would produce a portfolio of self-appraisal HTA and DVDs for each surgeon’s operations, which could be externally assessed if required.

We aim to continue the study and expand it to other laparoscopic and open operations. In addition to consultant surgeons, we aim to recruit registrar/resident surgeons and use the HTA as a possible teaching and learning tool for acquiring technical skills needed in performing a particular chosen operation. Finally, we aim to analyze and construct an HTA for assessing surgical theater team performance, evaluating how the HTAs for the key members of the theater team interact and influence each surgeon’s own HTA.