Keywords

1 Introduction

Training in robot-assisted minimally invasive surgery is costly [3] but also important in order to achieve the best possible outcomes [5]. In the training on actual robots, pointing and line drawing (so-called “telestration”) in the endoscopic view is often useful to support referential verbal communication by instructors (e.g., “look at this,” “cut here,” etc.) [8]. However, many surgical robots are operated using an immersive interface that blocks the visual communication between instructor and trainee (see Fig. 1). In these cases, one common approach is to overlay the endoscopic video image with the video image of a pointer and/or a line drawing and present the resulting video image to the trainee, who operates the robot, as well as the instructor, who controls the pointer [8]. One advantage of this solution is that the instructor or expert advisor (in general called “mentor”) does not have to be physically present but can be located at a large distance (so-called “telementoring”) [8, 11].

Fig. 1.
figure 1

Setup of a da Vinci S HD Surgical System. Left: surgeon (in our case the trainee) operating the console. Center: patient cart and assistant (in our case the instructor) using the telestration feature of a touch screen. Right: vision cart. Copyright 2015 Intuitive Surgical, Inc.

Full size image

In this work, however, we only consider the case of supervised training where the instructor is in the same room as the trainee who operates a da Vinci S HD Surgical System since this is the situation in the training courses at Aalborg University Hospital that are offered by our collaboration partner MIUC (Minimal Invasive Udviklings Center). The instructors are experienced surgeons and an experienced surgical assistant, who is often performing additional tasks (e.g., operating a suction device) while instructing the trainees.

The da Vinci S HD Surgical System offers the possibility to draw lines on a monoscopic touch screen and to overlay the stereo-endoscopic video image with a stereoscopic version of the line drawing [8]. This stereoscopic image includes an automatically added stereoscopic effect such that the drawing appears — to the trainee operating the robot — on a plane in front of the operating field. While this telestration feature works without affecting the resolution, frame rate or delay of the stereo-endoscopic image, the exact position of the drawn lines can appear ambiguous since they do not appear at the same depth as the tissue that the instructor points at. Specifically, the overlaid line drawing appears at different positions in the left and right image of the stereoscopic image and, in general, both positions are different from the position that the instructor touched on the monoscopic screen.

As reported by our collaborators and well-known in the literature [10, 12, 14], this ambiguity can make exact pointing very difficult. Furthermore, it is difficult for the surgical assistant to reach the touch screen while operating, for example, a suction device in the current setup of the training room. For these reasons, our collaborators usually do not use the telestration feature of the da Vinci S HD Surgical System in their courses. Another potential problem with a touch screen at eye’s height is arm fatigue [6]; however, this issue has not been mentioned by our collaborators.

In order to support exact pointing, we implemented a one-eyed cursor [14], which is controlled by a wireless gyroscopic air mouse, which can be held at a comfortable height to avoid arm fatigue [6]. One-eyed line drawings are supported by pressing a button of this air mouse. To overlay the stereo-endoscopic image with the image of the pointer and/or line drawing at the original resolution and frame rate without noticeable delay, we employ a recently proposed framework for hardware-based overlaying [7].

We are still adjusting details of the system based on the observed usage and feedback by instructors and trainees. So far, slightly different prototypes of our system were used and positively evaluated by five experienced instructors (including one experienced surgical assistant) in a total of four full-day training units in robot-assisted minimally invasive surgery on anaesthetised pigs.

The first main contribution of our work is to present the design and implementation of a one-eyed cursor for the da Vinci S HD Surgical System, which is comfortably controlled by a wireless, gyroscopic air mouse and does not affect the resolution, frame rate or latency of the stereo-endoscopic view; see Sect. 3. The second main contribution is the successful evaluation of a developing prototype of the proposed system in an operational environment, i.e., in actual training courses in robot-assisted minimally invasive surgery at Aalborg University Hospital; see Sect. 4. Before discussing these contributions, Sect. 2 reviews previous work.

2 Previous Work

Pointing at objects in stereoscopic images is basically a two-dimensional task, but it is usually considered a special case of pointing in three dimensions [10, 12, 14]. There appears to be a wide consensus that displaying a stereoscopic cursor at a different depth than the depth of the object that the cursor is pointing at should be avoided in order to avoid cursor diplopia (double vision). Instead, the cursor either should be displayed at the same stereo depth as the object or the cursor should only be displayed to one eye as first suggested by Ware and Lowther [14]. Schemali and Eisemann [10] observed better user performance with the first option and attributed this to the discomfort that a one-eyed cursor can cause (due to binocular rivalry). On the other hand, Teather and Stuerzlinger [12] observed — for certain pointing techniques — better user performance with a one-eyed cursor; in particular for objects far away from the screen depth. They attributed this to the problems of diplopia and accommodation-vergence conflicts, which do not occur with a one-eyed cursor.

In the case of stereo-telestration for robotic surgery, Hasser et al. [4] proposed to mark positions at the same depth as objects in the stereo-endoscopic image by computing a disparity map of the stereo-image. (See also Lamprecht et al. [8] and Zhao et al. [15].) Ali et al. [1] reported results of a user study with a prototype of such a system using a da Vinci surgical robot where three participants (“trainees”) had to identify pins that another participant (the “mentor”) pointed at. In comparison to 2D telestration, the trainees required significantly more time and committed non-significantly more errors with the three-dimensional marks. Similarly to the study reported by Teather and Stuerzlinger [12], these results might have been caused by diplopia and/or accommodation-vergence conflicts.

These works show that stereo-telestration at object depth for robotic surgery requires considerably more hardware and more complex software while impairing user performance — even if the software worked perfectly. Therefore, a one-eyed cursor appears to be an interesting and viable option to avoid the problems of stereo-telestration and at the same time retain the advantages of a stereo-endoscopic view.

Overlaying a stereo-endoscopic image with a computer-generated image usually results in a noticeable delay of more than 100 ms (e.g., [13]). Azuma et al. [2] state that delays as small as 10 ms can lead to a significantly worse user performance for certain tasks. This is consistent with results for low-latency direct touch which showed “noticeable improvement continued well below 10 ms” [9]. We assume that any noticeable delay (or reduction in frame rate) would reduce the user acceptance of our system.

An alternative to delaying the stereo-endoscopic image is to show it without delay side-by-side with a delayed image that is overlaid with another image. This approach is supported by the “TilePro” feature of da Vinci S HD Surgical Systems but it reduces the size and resolution of both, the original image and the delayed image with the overlay. Therefore, at least some surgeons appear to turn off this feature whenever possible [13]. Thus, we assume that any noticeable reduction in size or resolution of the stereo-endoscopic image would reduce the user acceptance of our system.

In order to overlay the stereo-endoscopic view with the image of a pointer without noticeable delay nor reduction of frame rate, size, or resolution, we employ a framework that we have recently presented [7]. Our specific usage of this framework is described in Sect. 3.

Hincapié-Ramos et al. [6] proposed a series of guidelines for the design of fatigue-efficient mid-air interfaces. In particular, they concluded that mid-air gestures at the height of the shoulder joint are more tiring than gestures between the height of the shoulder and the waist. Furthermore, they found that a clicking device for selection minimizes fatigue. Therefore, we assume that an air mouse that can be held at any height is more fatigue-efficient than a touch screen at eye’s height.

3 Proposed One-Eyed Cursor for Stereo-Endoscopy

To overlay the stereo-endoscopic video image of the da Vinci system with the computer-generated image of a pointer, we have employed our recently proposed system [7]. The core of the system is a desktop computer with two PCIe video cards (Blackmagic Design’s DeckLink HD Extreme 2), which is capable of overlaying the two channels of the stereo video image with any computer-generated imagery at full resolution and frame rate with less than 1 ms delay. We have also included a fail-safe system and a 3D TV and used a wireless, gyroscopic air mouse for user input to avoid arm fatigue [6] and to allow instructors to control our system from most positions in the training room.

To render the one-eyed cursor, the image of a pointer is only displayed to one eye by overlaying only one channel of the stereo video image. By default, the cursor is shown to the right eye, as this is the channel that the monoscopic displays of the da Vinci system default to. In some cases it is useful for the instructor to switch which eye the one-eyed cursor is displayed to, e.g., if the trainee is unable to perceive stereoscopic images or if the trainee is uncomfortable with a one-eyed cursor that is displayed to his or her non-dominant eye [14]. With our system, instructors can switch from one eye to the other by clicking the scroll-button of the air mouse. The console and the 3D TV that we introduced in the setup will then show the cursor to the other eye. In order for the cursor to show up also on the monoscopic displays of the da Vinci system, the instructor (or an assistant) has to change the channel shown on those displays by using the touch screen controls of the da Vinci system.

As described in Sect. 1, the telestration feature of the da Vinci system allows instructors to draw lines that are directly visible in the console. Due to the ambiguous position of the drawings, this feature has been rarely used in the training at Aalborg University Hospital in the past; however, our collaborating instructors are familiar with it and expected our solution to provide the same functionality. In our implementation (see Fig. 2), a green line is drawn from the tip of the pointer when the instructor presses and holds the left mouse button of the air mouse. We chose to use the color green based on observations and feedback from our collaborators, who stated that green is the least frequent color when operating on pigs and humans.

Fig. 2.
figure 2

The cursor and a line drawing overlaid on one of the video channels of the da Vinci S HD system. Note that a monoscopic image cannot convey the appearance of a one-eyed cursor.

Full size image

The telestration feature of the da Vinci system removes any line drawings when the endoscopic camera is moved. Alternatively, they can be removed by pushing a button on the touch screen. Initially, it was a user requirement that our system behaves similarly to the da Vinci telestrator, i.e., line drawings should stay on the screen until removed. However, when evaluating and regularly using the prototype, it proved more useful to have the drawings automatically disappear a few seconds after the instructor stops drawing. In this way, the instructors can keep the drawings on the screen as long as desired by holding down the drawing button, and there is no need for an additional button to remove the drawings.

To control the cursor, we have tested several wireless, gyroscopic air mice and found two candidates for the scenario at Aalborg University Hospital. We considered ease of use, precision, price, number of buttons, and ability to clutch, which is similar to lifting up a regular mouse to reposition it without moving the mouse cursor. The Gyration Air Mouse Elite was the most precise of the tested air mice, but it is also the most expensive one and still introduces some interaction problems (see Sect. 4). It has a “reverse clutch,” i.e., the user needs to hold a button on the bottom of the mouse to move the cursor. This turned out to be an intuitive clutching mechanism and also avoids unintended cursor movements, which would be distracting to the trainees.

The Measy RC9 Air Smart Mouse is a little less precise and offers a “toggle clutch,” i.e., the user has to press the same button to activate and to deactivate the control over the cursor. This turned out to be a less intuitive clutching mechanism and requires users to remember to toggle the clutch to avoid unintended cursor movements. The Measy RC9 Air Smart Mouse is significantly cheaper (less than one third of the price of the Gyration Air Mouse Elite), which might be important since the environment in which the air mice are used can be rough on electronic devices as fluids (blood, water, etc.) often get on the instructors’ hands when they are assisting. Waterproofing the air mouse by putting it into a plastic bag could protect it, but this would make it more difficult to use.

We have also investigated several other input methods, but based on initial testing they have proved either impractical in our setting (Kinect, LEAP motion) or simply too imprecise (Wii Remote).

4 Evaluation of Prototype in Operational Environment

Before evaluating a prototype of our system in training courses at Aalborg University Hospital, we observed several eight-hours training sessions without our system in order to assess the communication problems between instructors and trainees. The main conclusion from these observations was that the instructors did not use the telestration feature of the da Vinci system. Instead, they usually tried to rely on verbal communication and tended to take over the console of the robot when verbal communication alone proved to be insufficient. This approach was inefficient as considerable time was spent on unsuccessful verbal communication and taking over the console resulted in interruptions of the trainees’ operation of the robot and reduced their training time on the robot.

Since our collaborators were not actively using the telestration feature of the da Vinci system in their training courses, we decided against comparing it with our system in these courses since we are trying to interfere as little as possible with the courses. Furthermore, a comparative study between a one-eyed cursor and a stereo cursor at a different depth than the object that it is pointing at is very likely to confirm the previously published result that a one-eyed cursor is preferable in this comparison [12].

Therefore, we chose to evaluate a developing prototype of our system by installing it in the training room at Aalborg University Hospital and observing its impact on the training and the communication during the training. Moreover, we also observed and got feedback on the interaction with the air mouse and the perception of the one-eyed cursor and line drawings.

Our system has been in continuous use during the four most recent full-day training sessions at Aalborg University Hospital. Some of the interaction problems that were revealed in these sessions were fixed between sessions. For example, the instructors sometimes left the cursor in the middle of the screen without using it to point. To solve this problem, we hide the pointer when it has been in the same position for more than two seconds, as was also suggested by the instructors. Another improvement was to decrease the time before line drawings are removed from five seconds to two seconds after the instructors release the drawing button.

Of the two air mice that the instructors evaluated in the training courses, the Gyration air mouse was the preferred one due to its clutching mechanism, which appears to be the single most significant aspect of the usability of the air mice. As mentioned earlier, the reverse clutch helps to avoid unintended cursor movements and the instructors were able to use it immediately — presumably because it is similar to the clutching mechanism of the robot. The toggle clutch of the Measy air mouse proved to be unintuitive and was quickly abandoned by the instructors.

Our system clearly improved the visual communication from instructors to trainees, and with it, the communication overall. This was apparent by much more interactive communication between the instructors and trainees. The instructors used the cursor and line drawings to guide anatomical explorations by the trainees and to give task instructions, e.g., by pointing with the cursor and saying “cut here,” or drawing along a nerve and saying “the nerve is running here,” “grab here,” etc. — activities that previously often resulted in the instructors taking over control of the console.

We neither directly nor indirectly observed any need for switching the cursor to the other eye. None of the trainees reported any issues with perceiving the cursor and we did not observe any apparent miscommunication in relation to pointing. However, the way we implemented the switch caused some confusion as the instructors accidentally switched the channel in which the cursor was shown, causing them to lose sight of the cursor on the monoscopic displays. To avoid this, the instructors suggested that we make it more difficult to accidentally switch (e.g., by requiring to hold the button down for five seconds) and to add a message after the switch that tells them when the monoscopic displays are showing the channel without the cursor.

Generally, our system was evaluated positively by instructors and trainees. The trainees only had two complaints. First, the white part of the pointer was too bright which caused a slight flickering on the displays of the console. We have consequently changed the color to a bright gray. Second, the cursor was sometimes not hidden when it was left in the middle of the screen. This problem occurred because the Gyration air mouse can also be used as a regular mouse, which caused the mouse to unintentionally move when its proximity sensor was triggered such that our system assumed that the mouse was still in use for pointing. To solve this problem, we have blocked the infrared light that the mouse uses to measure distance.

In summary, the instructors found the prototype of our system very useful. In particular, they found it better and more precise than the telestration feature of the da Vinci S HD Surgical System. While the trainees never experienced the telestration feature of the da Vinci system, they benefitted from the improved visual communication with the instructors as compared to the training without any telestration system.

5 Discussion

The feedback that we received and the observed impact of the prototype of our system on the training in robot-assisted surgery at Aalborg University Hospital is very encouraging as it suggests that a one-eyed cursor that is controlled by a wireless air mouse can in fact improve the communication between instructors and trainees. However, we are fully aware that we are biased observers of our own system and that some of the instructors are similarly biased since they contributed to the development of the system. As most of the trainees have no prior experience with the da Vinci robot, their feedback cannot be used to compare our system with the telestration feature of the da Vinci system. Thus, further user studies are necessary to establish the benefits of our system once its development is completed.

6 Conclusion and Future Work

Based on the concept of a one-eyed cursor, we have developed a new telestration system for pointing and drawing in stereo-endoscopic views of the da Vinci S HD Surgical System. A prototype of the system has been integrated in training courses on robot-assisted minimally invasive surgery at Aalborg University Hospital and was positively evaluated by five experienced instructors.

Future work includes further observations and improvements of the system in regular use. This also includes improvements of the way the system is used by the instructors. For example, we assume that it would be beneficial to some trainees if instructors showed them the one-eyed cursor for each eye such that each trainee can choose the more comfortable alternative. Once the system and its usage are finalized, formal user studies are necessary to prove its benefits.

Furthermore, it would be very interesting to determine the level of discomfort that a one-eyed cursor can cause, the effect of eye dominance on this discomfort, and the percentage of affected users.

Our observations of the training with the proposed system showed that instructors still take over the robot in some situations, e.g., to demonstrate skills such as knot tying. Some instructors also ask trainees to look up from the robot, e.g., in order to communicate the best orientation of a needle with hand gestures. These situations could be addressed by overlaying the stereo-endoscopic view with a simulation of virtual robotic instruments that are controlled by instructors and displayed to trainees while they operate the console. Whether there is any advantage in displaying these virtual instruments to one eye only, is another open question.