Introduction

Laparoscopic liver resection is becoming more ubiquitous and high-volume centres could recently demonstrate a reduced complication rate and less perioperative morbidity even in major liver surgery compared to open surgery.1 Nevertheless, major laparoscopic liver surgery is still scarce in smaller centres mainly because of a flat learning curve resulting from lack of tactile feedback, bad depth perception and limited field of view as major drawbacks of the technique.2,3,4 The development of three-dimensional (3D) imaging in laparoscopic surgery in recent years improved depth perception as one of the major disadvantages of endoscopic surgery with monoscopic view.5 Its advantage over two-dimensional imaging could be shown in different studies.6,7 The introduction of 3D imaging improved the depth perception but was not the solution to overcome other disadvantages of the laparoscopic technique such as lack of tactile feedback, narrow field of view and limited capacity to handle intraoperative complications such as major liver haemorrhages when compared to open surgery. Those remaining drawbacks as well as the topic of vanishing lesions can be addressed by image guidance in laparoscopic surgery. Augmented reality (AR) has therefore great potential to enhance minimally invasive surgery. Unlike for neurosurgery, otolaryngology and orthopaedic surgery, where rigid structures facilitate a rather unproblematic registration, laparoscopic liver surgery faces the deformation problem of abdominal tissues and organs which results in a difficult registration procedure potentially requiring non-rigid registration techniques to achieve sufficient registration accuracy. Utilising three-dimensional imaging as a base for augmentation reduces the effort of the surgeon to percept and interpret possible overlay alignment errors and enhances depth perception.8 Augmented reality in combination with three-dimensional imaging in a clinical setting has been reported only in combination with robotic surgery so far.9,10,11,12

We therefore conducted this study to evaluate the technical feasibility and the clinical impact of a new AR system for laparoscopic liver surgery which combines the benefits of intraoperative 3D imaging (Einstein Vision 3.0, Aesculap AG, Tuttlingen, Germany) with an extended, commercially available IGS (image guided surgery) system (CAS-One AR, CAScination AG, Bern, Switzerland).

Materials and Methods

Patients

From January 2018 to March 2018, all patients with laparoscopic liver resection conducted with the laparoscopic AR-liver surgery system were included in this retrospective analysis. Patients were suitable for operation with the new system, if they fulfilled the following inclusion criteria: operation could be done in the same patient position like the preoperative imaging (generally supine position), availability of adequate preoperative imaging, smaller tumours with a predictable low operative conversion rate, consent for operation with the new system and analysis of their personal data. Only patients with preoperative imaging in the same patient’s position as the intraoperative position were included to minimise differences in the shape of the liver during operation.

Exclusion criteria were patients < 18 years of age and patients denying consent for analysis of their personal data.

The study was approved by the cantonal ethics board (KEK 2018-01009).

Pre-surgical Planning and Intraoperative Setup

Prior to surgery, 3D reconstructions of vascular territories and relevant structures like portal vein, hepatic arteries, hepatic veins and tumours based on preoperative computed tomography (CT) or magnetic resonance imaging (MRI) scans were obtained (MeVis Distant Service AG, Bremen, Germany). The processed DICOM (Digital Imaging and Communications in Medicine) files were then uploaded to be applied in the CAS-One navigation system.

Intraoperative setup is shown in Fig. 1. An EV 3.0 tower with a 32″ screen with pure (none augmented) 3D image was positioned to the right of the patient’s head and the augmented image on a 26″ screen was centred at the patient’s head together with the infrared tracking camera. A touch screen for the controlling of the navigation system was draped by a transparent, sterile plastic sheet and positioned to the left of the patient’s head. The touch screen was operated by the assistant standing on the patient’s left side, the primary surgeon stood between patient’s legs.

Fig. 1
figure 1

AR setup in the operating theatre

Full size image

Camera and Instrument Calibration and Image Registration

Camera and instrument tracking were accomplished by attaching retro-reflective spheres onto the laparoscope and a laparoscopic grasper (Fig. 2), which were tracked by a passive optical tracking system (NDI Polaris Vicra, Northern Digital, Canada). Camera calibration was conducted corresponding to the technique described by Zhang13 utilising a dedicated calibration unit incorporating a planar pattern observed by the camera at four different distances (Fig. 3). We used a stereoscope with 30° oblique optical axis for all procedures. The laparoscopic grasper was calibrated using a geometrical guide on the calibration tool. We used a rigid surface-based method for patient-to-image registration as first described by Herline et al.14 and evaluated in.15,16,17 Four points were chosen on the preoperative 3D model and matched to the actual liver by defining the same points using the optically tracked and calibrated laparoscopic grasper (Fig. 4).

Fig. 2
figure 2

Tracked laparoscope (a) and grasper (b)

Full size image
Fig. 3
figure 3

Calibration unit and intraoperative calibration of the laparoscope

Full size image
Fig. 4
figure 4

Landmark definition on the virtual model (a) and acquisition at the falciform ligament (b)

Full size image

Navigation Modes

There are two navigation modes available, an overview mode and a resection mode. The overview mode augments the 3D laparoscopic image with the complete preoperative reconstructed information.

As an option, the different types of vessels as well as liver segments and tumours can be selected for display (Fig. 5). The resection mode consists of a two row target at the tip of the tracked surgical instrument displaying the 3D live image in the centre and in the adjacent adjustable ring the augmented scene (Fig. 6). The desired radius of the displayed augmented structures can be selected.

Fig. 5
figure 5

Augmented reality overlay on live-laparoscopic image (left), virtual liver model with anatomical structures of interest (right)

Full size image
Fig. 6
figure 6

Resection mode visualising the AR overlay around the tool

Full size image

Surgical Equipment and Technique

A stepwise introduction of the stereoscopic imaging technique and the CAS-One AR system was chosen for a better evaluation of the benefit of augmented reality itself. 2D imaging was the standard technique for laparoscopic liver surgery in our clinic before introduction of the 3D laparoscope EV 3.0 followed by the CAS-One AR system after 2 months of practice. Patients were placed in lithotomy position with elevated right side in right posterior segment tumours. Single shot antibiotic prophylaxis was administered. We used a 3–4 × 12-mm trocar technique for all liver resections. Operation was started with mobilisation of the liver. Exposure of the hepatocaval confluence and the hepatic porta was a standard approach because these rather rigid areas are optimal for image registration. The further extent of mobilisation was defined by tumour localisation and kept to a minimum in order to prevent alterations in liver shape and consecutive impairment of registration accuracy. Registration of the preoperative 3D reconstruction to the laparoscopic scene was performed after mobilisation. In cases of bilateral tumour resection, the registration process was performed independently for each side. Intraoperative sonography could be used as a control measure for resection.

Resection was performed by an advanced energy device (HARMONIC ACE+, Ethicon Endo-Surgery, Inc., Cincinnati, USA) under additional laparoscopic ultrasound guidance.

Data Extraction and Analysis

Technical data were obtained from the CAS-One IGS-system’s log file and clinical data were collected from clinical records and collected in an electronic database. Registration accuracy was assessed as fiducial registration error (FRE).18,19 As a measure of intraoperative efficacy, the number of registration attempts and time spent on intraoperative calibration and registration were recorded. Clinical benefit was assessed trough a questionnaire which was completed by the primary surgeon after each operation. All measures utilised a 5-point Likert scale format ranging from 1 (high) to 5 (low).

Overall 30-day postoperative morbidity and mortality were graded according to the Clavien-Dindo classification.20 Severity of postoperative haemorrhage was defined after the grading by the International Study Group of Liver Surgery (ISGLS).21

Results

During the 3-month evaluation period of the IGS system from January to March 2018, 10 operations were performed comprising a total of 18 lesions. Four patients had one lesion, four patients two and two patients had three lesions. Patient and tumour characteristics are shown in Table 1.

Table 1 Patient and tumour characteristics
Full size table

All patients were operated by three surgeons composed of two consultants (conducting five procedures) and one fellow (five procedures).

In one patient, we did two separate registrations for the right and left liver lobe; therefore, 11 and not 10 registration processes in 10 patients were evaluated. Intraoperative parameters and calibration and registration parameters are indicated in Table 2. The setup (calibration) of the camera and instruments could be achieved in a median of 43 s (range: 29–174 s). Landmark definition at the preoperative image and definition of the equivalent points at the liver (landmark acquisition) counted for 53 s (range: 33–134 s) of the operation time per attempt. In cases of inadequate registration (> 10 mm), we conducted further registrations. Sufficient registration was achieved after four attempts and 8:50 min on average. The workflow of the operation was not significantly disturbed by the new system after registration.

Table 2 Operative parameters
Full size table

A planned right hepatectomy for an alveolar echinococcosis lesion had to be converted after 69 min to open surgery because a tumour extension was located along the left main bile duct which would have made a laparoscopical approach unsafe. Data of this patient have been used for technical analysis of the calibration and registration process only.

Measures of the system evaluation questionnaire are presented in Table 3. The feedback of the three surgeons was mainly positive. Especially, the ease of use of the system and the potential benefit for resection of vanishing lesions was scored very high. On the contrary, all surgeons consistently commented that the image registration for resection guidance is still too inaccurate, and even though they could see a probable benefit with further advanced technology in the future, they were not able to perform the actual resection relaying solely on the device.

Table 3 System evaluation questionnaire
Full size table

All patients’ admissions were on the day of operation. Median length of stay was 4 days (range 3–9). All patients received ward diet on the first postoperative day. Continuous epidural analgesia was administered for pain relief in one patient and a transversus abdominis plane block for the others.

Histology reported complete resection in all retrieved lesions. Four tumours have been removed with a resection margin < 1 mm of which three re-resections showed no tumour infiltration. One hepatocellular carcinoma was retrieved with a resection margin < 1 mm with no re-resection performed, but histology reports no vital tumour cells in this case according to the preoperatively performed trans-arterial embolisation because of spontaneous tumour bleeding. All other tumours have been retrieved with a resection margin > 1 mm.

Overall, no major intra- or postoperative clinical complications (Clavien-Dindo IIIb) occurred. While one patient received intraoperative transfusion due to mild haemorrhage, two patients had small bowel injuries during adhesiolysis and in the context of first port placement which were noticed and repaired immediately. Three patients developed postoperative Clavien-Dindo grade IIIa complications requiring an interventional drainage of a bilioma at the resectional surface, one patient was transfused with 2 units of red blood cells due to low haemoglobin levels postoperatively (post-hepatectomy haemorrhage Grad A).

Discussion

To the best of our knowledge we report the first clinical experiences in augmented 3D laparoscopic liver surgery.

Using AR in our pilot study did not change our standard surgical procedure and had little impact on the intraoperative workflow by adding only median 8:50 min of operative time for the instrument registration and image calibration process.

With a mean FRE of 9.2 mm, we achieved registration accuracy comparable to other systems. For their in vivo analysis, Thompson et al. used point and line landmarks to merge the preoperative model to the laparoscopic image. They measured the root mean square values of re-projection errors and reported an accuracy of 12 mm.22 Collins et al. showed that a rigid registration with raw data can be expected to have a target registration error of about 11 mm.23

In line with the observations of Ntourakis et al.24 and Huber et al.25 the application of our novel AR 3D system was found to be potentially helpful for the resection of vanishing lesions but not sufficient enough to lead the surgeon as a resection guide in complex cases where dissection close to relevant structures is mandatory. Because of its limited registration accuracy, the AR system was rather an additional navigational support in our operations than a reliable adjunct for the resection procedures. This fact is reflected in the use of intraoperative ultrasound control during and after resection in all surgeries. In our opinion, a registration accuracy of 10 mm is enough to resect vanishing lesions. These lesions being usually small (5 mm) can be resected with a 15-mm safety margin that includes the 10-mm registration accuracy without causing extensive tissue loss. Though a more accurate registration, we could resect with a smaller safety margin and extend the applicability of the method to larger lesions where 15-mm safety margin would not be tolerable. In order to achieve broader applicability of the AR navigation device, we would opt for more accurate registration in the range of 5–7 mm.

Utilising three-dimensional imaging as a base for augmentation reduces the effort of the surgeon to percept and interpret possible overlay alignment errors. Overlaying a 3D preoperative model to an intraoperative 3D picture allows the surgeon to easily judge potential registration inaccuracies. Advanced visualisation algorithms used in monoscopic image overlay applications to allow the surgeon a rapid identification of AR overlay errors as described by Thompson et al.22 are therefore not needed.

Every 3D virtual model is a static snapshot of the patient’s anatomy unless there is a dynamic registration process applied. With intraoperative live laparoscopic ultrasound merged with a stereoscopic video as described by Kang et al.26 or intraoperative CT imaging as described by Kenngott et al.27 the fusion between the virtual model and the intra-operative picture might be improved because of a minimal anatomical deformation between the acquisition times. The drawback of intraoperative imaging is its limited repeatability because of contrast agent application limitations and disturbance of the workflow. Nevertheless, in our view, it is not the limited initial registration accuracy which limits the applicability of the AR system but rather the tissue deformity because of liver manipulations, pneumoperitoneum pressure, and respiratory movements, and this problem can only be overcome with dynamic registration processes. In addition, tracked graspers as used in our “resection mode” add a further factor of imprecision to the augmented scene because of their tendency to bend. This problem could be addressed with electromagnetic tracking of the instrument tip as it has been studied by Kleemann et al.28 and Hayashi et al.29

The evaluation of the AR system with the questionnaire showed very consistent results over all three surgeons pointing towards its great potential for resection of vanishing lesions and support for demanding operations but also towards the prevalent insufficient fusion accuracy of the preoperative 3D model with the intraoperative scene as the chink of the system with the highest need for improvements. The qualitative rating of image overlay after registration shows a sufficient accuracy for resection guidance. Unfortunately, overlay accuracy is significantly degrading during the operation, especially after moving and grasping the liver tissue. In such cases, the overlay can be updated by repeating the registration process. In future work, we see a need to measure overlay inaccuracy in order to trigger registration updates. However, during the study, no quantitative measure of overlay accuracy was available.

In our clinic, we try to achieve surgical margin widths of at least 1 mm in every case and opt for even wider margins if surgically feasible. Margonis et al. showed in their meta-analysis comparing resection margins of > 1 mm to margins > 10 mm in colorectal liver metastasis resections a probable benefit for wider margins.30 In cases in which we did not achieve a 1-mm margin, we went for further resection.

As mentioned above, the goal of our study was to investigate efficiency, accuracy and the clinical benefit of a new image-guide system in 3D laparoscopic liver surgery. Due to the small sample size of this training cohort, a comparison to open or traditional laparoscopic surgery with regard to complication rates and oncological benefit was not possible.

Although our new system has a great potential to eliminate some shortcomings of other AR systems by introducing an augmented 3D laparoscopic picture, it is not yet technically mature enough to help surgeons guiding the resection in difficult cases. In our opinion, the main goal in future projects for augmented reality in liver surgery should focus on the development of dynamic registration processes. Further investigations regarding the clinical and oncological outcome of such a system will be performed once a non-rigid registration process is implemented with consecutively minimised overlay errors. Nevertheless, augmented reality in liver surgery has already the potential to help the surgeon visualise vanishing lesions or those invisible on conventional ultrasound and there is a great potential to act in future as a resection guide in difficult cases and might lower the threshold to use laparoscopic liver surgery instead of an open approach in major cases providing the benefits of laparoscopic surgery (as shown by Fretland et al.1) to even more patients. Furthermore, AR may have a high value in optimising the learning curve of surgeons approaching laparoscopic liver surgery.