Introduction

Assessment is an essential element in the evaluation of psychomotor skills during surgical training. Proficiency-based training and assessment implies that the trainee has to achieve a set of predefined criteria during their training to move to the next level in a safe and controlled learning environment [1, 2]. While proficiency-based training is practiced in many disciplines, for neurosurgery technical skills learning is linked to chronology, e.g., skills learned during the specific period of time spent in a residency program and in the operating room (6–7 years) in a novice/expert apprenticeship model [3–6]. Proficiency-based training ensures that specific criteria have been met while chronological-based training does not guarantee that a resident has achieved a degree of skill commiserate with competency in the surgical care of patients. Surgical competence is the ability of performing specific surgical skills successfully [7] and encompasses knowledge, technical, and social skills to solve familiar and novel situations to provide adequate patient care [8]. Interestingly, this definition focuses on “adequate” rather than “excellent” or “expert” patient care and what constitutes an expert in the field of surgery has not been clearly defined [9].

Cadavers, animal models, synthetic phantoms, manikin-based [10], and other mechanical simulators [11, 12] are all used for surgical training purposes. These models have the disadvantages of lack of realism, absence of disease-related pathology and bleeding and no limits on surgical exposure. The utilization of virtual reality (VR) simulators and appropriate metric technologies could address some of the shortcomings in the assessment and teaching of psychomotor skills. The ultimate goal of simulation-based training is to eliminate patient risks associated with technical skills learning. The learner achieves the desired learning outcome in a safe simulated environment where one can repeat the simulated procedure(s) with appropriate demonstrator and metric feedback. Various reports have been published on the application of VR simulators in different surgical fields including orthopedic [13], general [14], and laparoscopic surgery [15–17].

Development of neurosurgical VR simulators was first reported to simulate ventricular catheter insertion but has developed rapidly [9, 18, 19]. Several reports have also been published on VR simulation of neurosurgical endoscopic procedures [20–22]. A number of neurosurgical VR simulators have been developed for cranial procedures. Dextroscope, includes a neurosurgical planning system, called VIVIAN but does not allow manipulation of the brain tissue with force (haptic) feedback [23]. The ImmersiveTouch system integrates haptic feedback with a head and hand tracking system and is validated for ventriculostomy [24–27]. Measurement of the success rate in those studies is based on the results of the procedure, and specific metrics are not available to assess the operator’s psychomotor performance on other surgical tasks. The National Research Council Canada (NRC) supported by a research group of surgeons has developed a VR simulator, called NeuroTouch, to simulate neurosurgical procedures including brain tumor resection [9, 22, 28–30]. NeuroTouch is based on a finite element method and can simulate brain deformations, uses real-time computing to assess simulated brain tumor and normal brain tissue removal, and can generate and measure bleeding. A set of virtual reality tasks are defined on NeuroTouch, including, (i) ventriculostomy, (ii) endoscopic nasal navigation, (iii) brain tumor resection, (iv) hemostasis, and (v) microdissection [28–30].

During an intracranial tumor resection, the goal of the operator is to resect the brain tumor with instruments using techniques and applying forces that adequately remove the brain tumor but result in minimal injury to surrounding normal brain tissue. However, the technical and cognitive aspects necessary to accomplish this goal by the expert neurosurgeon are not totally understood [9]. Since it is difficult to accurately measure the range of psychomotor skills employed by the expert in the operating room, assessing and/or imparting these skills to the novice/resident doing a brain tumor resection can result in errors which can impact patient safety.

The NeuroTouch platform provides two specific Tier 1 metrics during a simulated brain tumor resection: percentage of brain tumor resected [30] and volume of simulated “normal” brain tissue surrounding the tumor removed. Tier 2 metrics include duration of time taken to resect the brain tumor, instrument path length, pedal activation frequency, and sum of applied forces [30]. Tier 3 metrics are novel metrics derived from force pyramid and force histogram.

The specific objectives of this pilot study were: (1) to assess whether these three sets of metrics, extracted from the NeuroTouch data, can encompass the range of novice/expert performance seen during the resection of a wide variety of simulated brain tumors (2) to evaluate whether the performance of a novice resident (second year) and an expert neurosurgeon can measure the safety, quality, and efficiency of technical performance during the resection of simulated brain tumors. Our results demonstrate that the three sets of metrics studied do encompass the wide variability of technical performance observed during novice/expert resections of simulated brain tumors and are useful to quantify the safety, quality, and efficiency of these resections. Tier 3 novel metrics including data derived from the force pyramid and force histogram were consistently able to differentiate novice from expert performance. Based on our results, the assessed metrics appear to be appropriate to be incorporated into surgical research paradigms designed to improve the understanding of how surgical performance is dependent on both psychomotor and cognitive input.

Methods

First, an overview of NeuroTouch, the simulator used in this study is provided. Second, we describe the simulation scenarios and the operative resection procedure, and lastly, we introduce the three sets of metrics that were employed to assess psychomotor skills of novice/expert operators in the simulated brain tumor resection tasks.

NeuroTouch

NeuroTouch provides a trainee with the opportunity to practice surgery in a virtual environment. The physical tools held by the trainee can be observed and tracked in the form of virtual tools which interact with virtual tissues providing haptic feedback. The detailed description of the NeuroTouch platform has been reported previously [28].

Figure 1a, b demonstrates the main hardware components of NeuroTouch. Various physical tools can be used to perform different simulated operations. Two tools can be held simultaneously, one in each hand. The physical shape, size, and behavior of these tools are similar-to-real surgical tools. Some tools such as the simulated ultrasonic aspirator are activated by a foot pedal. A display depicts the virtual operating scene together with the virtual tools which correspond to the physical tools in the hands of the trainee. This depiction is in the form of two images that are used by a stereoscope to generate a 3D visualization [31]. The fused two images can be viewed on an auxiliary display. The stereoscope simulates the neurosurgical microscope used in the operating room to provide a 3D magnification of the field. A haptic device is connected to each tool which provides force feedback corresponding to the interaction between virtual tools and tissues. From the haptic device, one could obtain the force that the operator applies with the tool on the virtual tissue and also the tool tip position. A computer runs the simulator software, which includes graphics, haptics, and tissue mechanics.

Fig. 1
figure 1

Outline of the components of the NeuroTouch Surgical Simulator (a, b) and a virtual craniotomy scene (c)

Full size image

Figure 1c displays a craniotomy simulation in NeuroTouch to teach handling the ultrasonic aspirator. In this example, the goal is to use a virtual ultrasonic aspirator to remove a virtual brain tumor with minimal removal of the virtual “normal” surrounding brain tissue. The ultrasonic aspirator, the normal brain tissue, and the brain tumor can be identified in Fig. 1c. The simulator can also simulate bleeding resulting from rupture of brain tumor and normal blood vessels. A bleeding source is visible in the resection cavity that is created by the ultrasonic aspirator in the center of the brain tumor.

Simulation scenarios

In this pilot study to more accurately assess the ability of performance metrics to encompass the wide variety of brain tumor stiffness and colors seen during surgical procedures, it was considered important to develop a series of simulated brain tumors which would be an initial approximation of the variety of human brain tumors. To both understand and represent the range of human brain tumor stiffness in our scenarios, we initially measured multiple samples from seven human brain tumors immediately after removal at operation and found that the stiffness of these brain tumors varied from Young’s modulus: 0.5 kPa to Young’s modulus: 15 kPa [28]. A range of colors were chosen for the scenarios including black (simulated malignant melanoma) to give maximal color difference between the background “normal” tissue and the brain tumor, simulated brain tumor appearance (glioma) and white (white matter with or without tumor infiltration). Figure 2 outlines the 6 scenarios that were considered an initial approach to the goals of representing the wide range of brain tumor stiffness and colors that a resident and/or a consultant may face during operative procedures. In Scenarios 1 through 3, each scenario comprised three brain tumors with the same visual properties but different stiffness to assess the influence of this tumor property on surgical performance. All simulated brain tumors were black in Scenario 1, with simulated brain tumor appearance in Scenario 2, and similar-to-background white in Scenario 3. In each scenario, the stiffness of the brain tumors was either “soft” (Young’s modulus: 3 kPa), “medium” (Young’s modulus: 9 kPa), or “hard” (Young’s modulus: 15 kPa). The stiffness of the background “normal” tissue was the same as that of a soft brain tumor: 3 kPa. Scenarios 4 through 6 each comprised three brain tumors of different colors but the same stiffness to include tumor color as a variable, namely soft brain tumors in Scenario 4, medium stiffness brain tumors in Scenario 5, and hard brain tumors in Scenario 6. Each of these scenarios included the three visual properties, namely black, simulated brain tumor appearance, and white.

Fig. 2
figure 2

Visual appearance of the 3 simulated brain tumors in each of the six individual scenarios which made up the 18 brain tumors assessed in each of these studies. The stiffness of each tumor is designated as soft (3 kPa), medium (9 kPa), and hard (15 kPa) and tumor colors outlined as seen by the operator in each of the six scenarios

Full size image

A number of configurations were considered for the simulated brain tumors. However, three ellipsoidal tumors extending underneath the normal tissue surface were utilized to standardize both the tumor shape and the complexity of the resection. This shape allowed us to increase the difficulty of the task and provide a more realistic representation of human brain tumors (some glial tumors and meningiomas grow underneath normal brain tissue) while allowing for the accurate continuous determination of instrument position during the resection. Figure 3 presents a side view segment that is protruding above the “normal” tissue plane. The projection of this segment on the “normal” tissue plane is denoted as R1 and contains the bulk of the brain tumor tissue which tends to be removed first. The projection of the brain tumor segments that extend underneath the “normal” tissue plane is denoted as R2, and the operator must carefully remove this component of the tumor resulting in deformation of both the tumor and the “normal” surrounding tissue similar to what can be seen during the operations on human brain tumors. The projection of the “normal” tissue segments that surround the brain tumor where it has the largest diameter is denoted as R3. Online Resource 1 demonstrates how an operator can remove the series of simulated brain tumors with the simulated ultrasonic aspirator.

Fig. 3
figure 3

Lateral view of the brain tumor geometry and elliptical shape utilized in each scenario demonstrating the three identical tumors, tumor buried underneath simulated “normal’ tissue and the R1, R2, and R3 regions studied individually

Full size image

Operative resection

Before entering the study, each participant signed a consent approved by the McGill University Research Ethics Board. The goal was defined for the operators as removal of the three ellipsoidal brain tumors with minimal removal of the background tissue, representing simulated “normal” brain tissue. This was to be accomplished one tumor at a time, in a predefined sequence using the simulated ultrasonic aspirator in the dominant hand until all 18 brain tumors were resected. At the beginning of the test, the operators were given one practice scenario to acquaint themselves with the procedure. All tumors in the practice scenario had medium stiffness and were visually similar to the simulated brain tumor appearance seen in Scenario 2. The practice scenario was not used to assess performance, and the operators were not aware of the metrics to be used. The ultrasonic aspirator was activated by a mimicked foot pedal and used at constant suction and ultrasonic power to standardize this component of the study. Figure 4 demonstrates a schematic of each scenario. First, with the tip of the virtual ultrasonic aspirator, the operator touched a virtual START button to start the simulation. Then, the operator removed Tumor 1, followed by touching the virtual STOP button using the tip of the virtual ultrasonic aspirator. The procedure was repeated for Tumor 2 and Tumor 3. The operators were given 3 min to remove each tumor, followed by a 1-min break before proceeding to the next tumor. The 1-min break was mandatory, whether the operators finished the tumor removal in 3 min or less.

Fig. 4
figure 4

Schematic of a model simulation scenario outlining how the operator initiated the simulation and the sequence of brain tumors to be removed

Full size image

Metrics

A series of metrics were assessed during the performance of the junior resident and the expert neurosurgeon in the simulated scenarios previously outlined. The concept to be assessed was that the series of metrics utilized encompass the range of novice/expert performance on the simulated brain tumor resection tasks and can differentiate novice from expert performance. Preliminary experiments had demonstrated that each simulated brain tumor could be completely removed within 3 min. It is understood that operations to resect human brain tumors are much longer and often only a partial removal is possible and concomitant radiotherapy and chemotherapy may or may not control further growth. However, the tumor size and a maximum time period of 3 min for each brain tumor resection were chosen to force the operator to deal with the complex problem of the tumor-surrounding simulated “normal” tissue interface. The operator psychomotor–cognitive interplay, when resecting the brain tumor at this interface, was considered an important focus of assessment which simulation allowed us to explore.

Tier 1 metrics are those provided directly from the NeuroTouch platform after the completion of each simulated brain tumor resection, namely percentage of brain tumor resected and the volume of simulated “normal” brain tissue surrounding the tumor removed.

Percentage brain tumor resected

This metric provides a measurement of the percentage of the simulated brain tumor removed. The goal of the operator is to resect the maximum amount of the simulated brain tumor without removing any of the surrounding simulated “normal” brain tissue.

Volume of simulated “normal” brain tissue removed

This metric provides a volume measurement in cubic centimeters (cc) of simulated “normal” brain tissue resected during the brain tumor removal. The goal of the operator is to remove no “normal” brain tissue during brain tumor resection and is a measure of unwanted tissue injury during the resection and thus an inverse measure relating to patient safety.

Tier 2 metrics are metrics generated from the NeuroTouch data. These are the instrument path length, the duration of time taken to resect the brain tumor, the pedal activation frequency, and the sum of applied forces.

Instrument tip path length

The length of the path traversed by the tip of the tool is used as a metric to measure the efficiency of the tool usage during the brain tumor resection. The goal of the operator is to carry out the resection using the most efficient and safest path trajectory of the ultrasonic aspirator. The specific trajectories, average, and maximum velocities related to the tool tip tracking could also be scrutinized to quantitatively assess the speed of instrument maneuver used by each operator as well as capture operator tremor or hesitation.

Duration of time taken to resect the brain tumor

The amount of time that it takes to resect a brain tumor was used as an efficiency metric. This was the active duration that the tool was in contact with the brain tumor and normal brain tissue excluding the idle times during the task. The operator must use cognitive input during the resection of the brain tumor to be efficient but not to compromise patient safety by removing “normal” brain tissue during the simulated procedure.

Pedal activation frequency

The number of times an operator activates the ultrasonic aspirator with the foot pedal was assessed by this efficiency metric. To maximize the efficiency of the ultrasonic aspirator, the operator should employ it utilizing the minimum number of pedal activations.

Sum of applied forces

The sum of all applied force samples measured during the simulated operation was used as a measure of the overall applied force employed to remove the brain tumor. The goal of the operator was to apply the minimal forces necessary to safely resect the brain tumor.

Novel Tier 3 metrics analyze the operator performance in specific detail. These metrics were designed to provide spatial and frequency analyses of force values. A graphic representation of force values versus xy coordinates of the tool tip provides a spatial analysis of forces applied in various brain tumor regions. We call this representation a force pyramid, since an experienced operator would apply appropriate forces in the safe middle portions of a brain tumor and smaller forces in the unsafe areas near or outside the border between the brain tumor and the “normal” brain tissue. Based on Fig. 4, three metrics can be defined as (1) the sum of forces applied in regions projected to R1, (2) the sum of forces applied in regions projected to R2, and (3) the sum of forces applied in regions projected to R3.

Sum of forces applied in regions projected to R1

This metric provides a safety measure of an operator’s force application in the specific central brain tumor region. An experienced operator would use forces designed to maximize safe brain tumor resection in this region.

Sum of forces applied in regions projected to R2

An experienced operator would remove the brain tumor in these areas more carefully possibly utilizing a set of appropriate but smaller forces.

Sum of forces applied in regions projected to R3

This metric provides a measure of how much force the operator has applied on the normal tissue surrounding the brain tumor. The goal of an experienced operator would be to use minimal or no forces in these regions.

A force histogram provides an analysis of how frequently various force values have been utilized. It is important that the operator applies a safe amount of force on tissue continuously during the resection. Although the sum of applied forces provides an overall value concerning the applied force, it does not provide information about various force values and the frequency at which these forces were applied. A histogram provides the percentage duration that an operator spent in various force zones. Various metrics could be derived from the force histogram. In this first-time application of the force histogram, we extract a force bandwidth.

Force bandwidth

We provide a preliminary definition for the force bandwidth as the band that is delimited by the smallest and the largest force values that have been used at least once by the operator. A more experienced operator could resect the brain tumor with a smaller force bandwidth.

Results

In this section, we provide the results obtained from the two operators utilizing the series of metrics previously outlined focusing on assessing the usability of these metrics for training rather than a comparison of the skills of a resident and experienced neurosurgeon. The neurosurgical program in Canada is a 6-year program (PGY1-6), and the results used in this paper are from a resident in the second year of training (PGY-2), who had been exposed to 6 months of neurosurgery before participating in this study. The expert neurosurgeon had significant experience in the resection of intracranial tumors. In the results to be presented, the brain tumor types (visual and tactile properties) have been abbreviated according to Table 1. For example, a black hard tumor is represented as “BH” in the graphs.

Table 1 The abbreviations used for each of the 18 brain tumors studied during this pilot study outlining their visual and tactile properties
Full size table

Percentage brain tumor resected

Figure 5a provides the results of the percentage brain tumor removed by the two operators. The percentage brain tumor resected by the surgeon was greater or equal to the percentage brain tumor resected by the resident in 12 of 18 simulated brain tumors. In four of the simulated brain tumors with “soft” stiffness (Young’s modulus 3 kPa) and/or white color the surgeon removed the brain tumor incompletely while the resident did a complete removal suggesting that the resident may not have appreciated the complexity of removing these more difficult tumors without substantial injury to surrounding tissues. These results suggest that brain tumor regions with low stiffness and white color such as tumor infiltrated white matter may be problem areas for resident performance and further studies seem warranted to understand these differences in cognitive performance when dealing with these brain tumor regions.

Fig. 5
figure 5

Tier 1 metrics (a) Percentage brain tumor resected by the surgeon and the resident (b) “normal” brain volume removed in cc’s by the surgeon and the resident during the resection of each of the 18 tumors

Full size image

Volume of simulated “normal” brain tissue removed

Figure 5b provides the results of the “normal” brain volume removed by each operator during removal of brain tumors. The resident removed more of the “normal” brain tissue in 17 of 18 simulated brain tumors consistent with the conclusion that the resident was far less able to remove the simulated brain tumors independent of their stiffness and/or color without removing more surrounding “normal” tissue. The interface between the brain tumor and the surrounding tissue appears to be one region to focus on during resident training since this region may be a problem area for resident performance.

Instrument tip path length

Figure 6 demonstrates a comparison of the tool tip trajectories between the surgeon and the resident. The tumor in this example is the first tumor of Scenario 6, i.e., the black medium tumor. Figure 6a demonstrates the tool tip trajectory for the surgeon in xy and xz coordinates. Figure 6b demonstrates the tool tip trajectory for the resident in xy and xz coordinates. Comparison of the trajectories in xz coordinates reveals that the resident makes more vertical movements of the instrument to remove the tumor. The resident also utilized a larger 3D space during the task. Online Resource 2 demonstrates tool tip tracking during resection of the black medium tumor in Scenario 6 by both operators.

Fig. 6
figure 6

The actual simulated ultrasonic aspirator tool tip trajectories utilized by the surgeon and the resident to resect a black medium stiffness brain tumor in scenario 6 in the xy (top view) and xz (lateral view) coordinates

Full size image

Using the trajectories such as those in Fig. 6, one can calculate the path length for the tool. Figure 7a presents a comparison of path lengths corresponding to all tumors for the two operators. This figure shows that the surgeon resected the tumors with a shorter path length in the majority of the tumors (13 of 18 studies).

Fig. 7
figure 7figure 7

Tier 2 metrics (a) Instrument tip path in mm (b) durations of time taken to resect the brain tumors in sec (c) pedal activation frequencies (d) sum of applied forces in Newtons (N) of surgeon and resident performance during the resection of each of the 18 brain tumors

Full size image

Duration of time taken to resect the brain tumor

Figure 7b presents the results of the amount of time that each operator spent to resect each tumor. The resident took longer to resect every tumor studied. Although a lesser time taken to complete an operative procedure may not be directly related to patient outcome, it would appear that the surgeon is able to remove these simulated brain tumors in a shorter time then the resident for all tumors studied. This is consistent with the conclusion that the surgeon is using a set of psychomotor skills resulting in improved efficiency of brain tumor resection. These data suggest that an increased emphasis on the reasons for the decreased speed of operation using the ultrasonic aspirator by residents is one component of resident teaching that could be further investigated and optimized.

Pedal activation frequency

Figure 7c presents a comparison of pedal activation frequencies for the two operators. For the surgeon, this metric is equal to one, meaning that the tool has been kept activated during the entire operation for 17 of 18 tumors. Since the resident activated and deactivated the pedal multiple times during removal of each tumor, this may have been one factor that contributed to increased duration of resident operations which would have decreased operator efficiency. If this is a consistent finding in further studies and the reasons for this behavior are identified, this behavior could be easily modified.

Sum of applied forces

Figure 7d presents a comparison of the sum of applied forces by the ultrasonic aspirator for the two operators. In all cases, the applied force sums are larger for the resident. This demonstrates that the resident always used increased forces to resect the tumors. However, the tumors could have been removed with less force as demonstrated by the results of the surgeon.

Figure 8 displays a comparison of force pyramids for the surgeon and the resident when removing the black medium tumor in Scenario 6. Each spike at a particular xy coordinate represents the sum of applied force values in all regions with the same xy coordinate. The color of the surface also reflects the force value in logarithmic scale. Each force pyramid has been coupled with a top view color map of the force pyramid in logarithmic scale which provides a better regional analysis of force application. In these color maps, regions R1, R2, and R3 (as defined in Fig. 4) can be distinguished by the area within the circle outlined by the solid white line, the area between the solid white and dashed white lines and the area outside of the dashed white line, respectively. As this figure shows, the surgeon has consistently used less force values in different regions. Online Resource 3 demonstrates these pyramids from different angles. Figure 9 presents the three metrics that are extracted from different pyramid regions.

Fig. 8
figure 8

Force pyramids (3D view and top view) for the surgeon (a) and the resident (b) obtained for the resection of a medium stiffness brain tumor in Scenario 6 demonstrating applied forces in Newtons (N) in the xy and xz coordinates. The color map on the left outlines the colors corresponding to different forces in Newtons (N)

Full size image
Fig. 9
figure 9

Spatial analysis of sum of forces in Newtons (N) in (a) applied in regions projected to R1 (b) applied in regions projected to R2 (c) applied in regions projected to R3 during the resection of the 18 brain tumors by the surgeon and the resident

Full size image

Sum of forces applied in regions projected to R1

Figure 9a outlines the sum of applied forces in the central area for each tumor. The applied force sums in central areas are smaller for the surgeon than the resident for all tumors. These results are consistent with the concept that a surgeon has learned to use consistently lower forces during tumor resection even in central relatively safe areas of tumors which may be related to both increased psychomotor skills and concerns related to safety.

Sum of forces applied in regions projected to R2

Figure 9b outlines the sum of applied forces in the ring shaped border region. The applied force sums in border areas are smaller for the surgeon than the resident for all tumors. These data also emphasize that the surgeon can resect the tumor underlying simulated normal tissue regions with decreased forces as compared to the resident again outlining an emphasize on safety which may be one reason for the decreased volumes of surrounding “normal” tissue removed by the surgeon seen in Fig. 5.

Sum of forces applied in regions projected to R3

Figure 9c outlines the sum of applied forces in the surrounding “normal” tissue areas. The applied force sums in outlying areas are smaller for the surgeon than the resident for all tumors again consistent with the concept that the surgeon focuses consistently on safety in all three of the regions assessed.

This initial pilot study demonstrates that in 18 tumors with 3 different levels of stiffness and 3 different colors that in every case the resident used more sum of forces than the surgeon in all three regions studied with wide variations in sum of forces related to tumor stiffness and/or tumor color. The surgeon tended to use very low and constant sum of forces for all tumor regions irrespective of tumor stiffness and/or color compared to the wide variation of sum of forces used by the resident. These initial data suggest that the sum of forces applied in the tumor and the surrounding tumor interface may be particularly different when residents and surgeons performance is assessed. This has generated a series of new hypotheses that are being tested.

Force bandwidth

Figure 10 displays a comparison of force histograms for the surgeon and the resident when removing the black medium tumor in Scenario 6. Each force histogram is coded into a color map in logarithmic scale. The color maps are particularly useful when comparing several histograms at the same time. Figure 11 demonstrates a comparison of all histogram color maps corresponding to removal of all tumors by both operators. This figure shows that the surgeon consistently uses a range of lower forces.

Fig. 10
figure 10

Force histograms and force bandwidths (arrows) with corresponding color maps outlining the percentage of time (%) that a particular force was employed to remove the black medium stiffness brain tumor in scenario 6 by the surgeon and resident. The % of time that force was used is outlined by the blue line

Full size image
Fig. 11
figure 11

Force histogram color maps for each of the 18 brain tumors resected by the surgeon and the resident in Newtons (N). The corresponding color map outlines the percentage of time (%) that a particular force was employed to remove each brain tumor

Full size image

Figure 12 presents the force bandwidths for each operator to resect the individual tumors. The surgeon utilized smaller force bandwidth in all cases. The maximum force bandwidth is also a measure of the maximum force utilized by the operator during the resection. Surgical errors may result during sudden planned and/or unplanned increases in instrument force bandwidth.

Fig. 12
figure 12

Force bandwidth in Newtons (N) which is equivalent to the maximum force applied by the surgeon and the resident during the resection of each of the 18 tumors

Full size image

At present, when a surgeon is operating or supervising a resident during an operation, there is no method for the surgeon to know the exact forces being applied by an instrument to a tumor and/or the surrounding brain. A surrogate used to assess improper force application is the brain injury that results from the utilization of too much force over time and/or the use of a sudden inappropriate force. This resultant brain injury may involve direct trauma to the brain and/or cranial nerves, increased brain edema and/or vessel damage. Many of these operative errors may result in increased patient morbidity and mortality [32]. Preventing these types of errors by training in a simulated environment could have a significant impact on patient safety and improved patient outcomes. Conceptually, the operator during a simulated operation could be provided with a visual output of his/her force pyramid and force histogram either continuously during and/or at the end of any type of simulated surgical procedure. This individual force pyramid and force histogram could then be easily visually compared to that of an “expert” and forces adjusted and readjusted in subsequent simulated operative procedures to attain predefined goals. Although the force pyramid and force histogram are useful methods for providing the operator with immediate visual qualitative information concerning the forces and frequency that these forces are applied in multiple diverse brain tumor regions, they do not provide quantitative information. An understanding of the actual forces applied in specific domains such as the R1, R2, and R3 regions, outlined in these studies, may allow the operator to further modify force application by any instrument and develop psychomotor skills memory that keep him/her away from dangerous predefined operative danger zones. A simulator has the potential to provide immediate warnings when a force is being applied outside of standardized benchmarks for that procedure and/or tissue damage is ongoing in specific brain regions and thus can maximize psychomotor learning and hopefully improve patient safety.

Discussion

Performing technical skills in the operating room is a stressful and complex experience, resulting in variable responses based on the operative situation that is being addressed. Reznick and MacRae [6] comment that surgical educators need to develop and use objective and meaningful assessment tools that are reliable and valid to assess trainee surgical skill acquisition and progress. Specific assessment tools may be reliable and valid for basic surgical skills acquisition, but these tools may not be useful when utilized to measure the skills needed to perform difficult neurosurgical operations such as the brain tumor resection [9, 33–35]. There are no validated assessment tools to measure technical skills performance for complex neurosurgical procedures. Consequently, different neurosurgical residency training programs use different means to assess trainees’ technical skill acquisitions [9]. The assessment of the psychomotor skills of a resident by a series of consultant surgeons is subjective and by its nature incomplete. Validated objective criteria for the technical skills assessment of surgical competence are not included in the neurosurgical curriculum since these criteria have not been developed. The development of metrics as demonstrated in this study may help develop such criteria.

The goal of this pilot study was to assess whether a series of metrics derived from the NeuroTouch simulator platform can measure the range of psychomotor skills involved in the resection of simulated brain tumors. Three tiers of metrics which can be objectively assessed and based on data obtained from the NeuroTouch simulator can encompass and have been utilized to quantify the range of performance, seen in resident and expert performance during simulated brain tumor resection. These metrics need to be validated and can be modified further after analyzing the performance of a large number of residents and surgeons [30]. At present, there are fifteen members of the NeuroTouch Consortium spread across three continents and a critical aspect involving these centers is the standardization of performance metrics. The development of standardized metrics would allow the accurate comparison of data obtained from all centers assessing the NeuroTouch platform in validation studies.

We defined three tiers of metrics dependent on the ease with which these metrics can be derived from the NeuroTouch platform and to allow an assessment of the quality and efficiency of the operation and multiple measures of the most important components of any brain tumor operation—its safety (Fig. 13). Since data have demonstrated that 1 out of every 150 patients hospitalized patients dies secondary to a complication and 40 % of these problems occur in surgical patients [32, 36] with half of these felt to be preventable [37] a strong focus on operator safety in the development of metrics appears warranted. Metrics designed to assess, evaluate, and teach safe surgical psychomotor skills should have high priority in any research validation program. These metrics will also allow for accurate comparison with other VR simulation platforms that may be developed to assess brain tumor resection and other neurosurgical procedures. The NeuroTouch platform provides a direct readout of percentage of brain tumor resected which is a measure of quality of operator performance and volume of normal simulated tissue removed which is a critical component of the safety with which the procedure is carried out (Fig. 13). Although not assessed in these studies, the volume of bleeding in cc is also measured directly by NeuroTouch. While these metrics are useful, they do not encompass all the information necessary to help us understand how an expert neurosurgeon performs complex brain tumor operations. Tier 2 metrics are not provided directly by NeuroTouch but can be derived from the data stored by the system during a simulated procedure. Duration of the procedure, instrument path length, and pedal activation frequency are measures of operator efficiency, while sum of applied forces evaluates another measure of operator safety. Tier 3 contains novel metrics which can also be generated from NeuroTouch data files. Force bandwidth and data derived from specific brain tumor regions of the force pyramids may be particularly useful metrics to explore operator performance and safety. The forces applied to remove the brain tumor at the interface between the brain tumor and normal tissue may be especially important to focus on in further research since the expert may use a learned but measured complex psychomotor–cognitive interaction to obtain the desired surgical result in this difficult brain tumor environment. Imparting this complex information to the trainee would appear to be an appropriate goal of any surgical training program.

Fig. 13
figure 13

Organization of metrics used in this study based on safety, quality, and efficiency metrics

Full size image

These metrics can also be used to explore a series of issues relating directly to the question of neurosurgical expertise. First, how do “expert” neurosurgeons actually perform neurosurgical operations? Questions such as what tactile, visual, and/or other cues along with what forces are employed by expert neurosurgeons during the technical components of their operations are presently being studied using the three tiers of metrics available utilizing NeuroTouch technology. Second, with the proper metrics, tools, and curriculum, can VR simulation impart specific technical skills to residents helping to shift the goal of neurosurgical training programs from teaching to competence to teaching to expert level? To answer these questions, the metrics employed in simulator assessment need to accurately assess the range of techniques employed by a novice and expert operator. This information would improve our understanding of how and when specific operator performance is dependent on psychomotor ability and/or cognitive input.

The results of this pilot study need to be interpreted with caution. First the study is only a pilot study and provides only the data from one novice and one expert. Studies are presently being carried out including large numbers of medical students, residents at various levels of training, and neurosurgical staff to evaluate the validity of these metrics. Second, in these studies, the operator was only allowed to use the dominant hand holding the simulated ultrasonic aspirator for the tumor resection. To be more realistic, a scenario with tumor-associated bleeding controlled by a sucker in the non-dominant hand and the ultrasonic aspirator in the other hand and involving bimanual psychomotor activity would need to be evaluated [30]. Third, the short duration of the task and its level of difficulty may not have been able to accurately discriminate the levels and/or quality of performance among the two operators. Fourth, the performance of each participant was not videotaped and could not be assessed by a standardized assessment scale since no such scale has been developed for neurosurgery. In bariatric surgery, the videotaped assessment of greater surgical technical skills is associated with lower rates of complications [38].

After the standardization and validation of these metrics and other metrics, it will be essential to determine benchmarks for each individual metric [35, 39]. Based on these benchmarks, thresholds could be determined that would help develop training curriculum for residents. The benchmarks could also be used for self-assessment and trainees could assess themselves and improve their performance using validated metrics as a guide. For example, resident operators could compare their tool tip trajectory (Fig. 6b) with a benchmark and modify their excessive maneuver in the xyz planes. Trainees could also compare their force pyramid and force histogram with a benchmark to investigate whether the forces that they apply are safe for patients in the operating room.

Conclusions

In this pilot study, we have explored three sets of metrics derived from NeuroTouch, an advanced VR simulator, developed by the NRC, which has 3D realism and haptic feedback. These metrics were obtained for a variety of simulated brain tumor resections, performed by a junior resident and an expert neurosurgeon and show that these metrics encompass the range of novice/expert performance on the simulated tasks. The metrics, assessed in this study, can also be utilized to further assess the quality, efficiency, and safety of simulated operative procedures as outlined in Fig. 13.