Laparoscopic surgery underwent a rapid development since the first laparoscopic cholecystectomy, which took place more than 25 years ago [1]. Even-though evidence shows that laparoscopic liver resection (LLR) reduces intra-operative bleeding, hospital stay, postoperative morbidity and pulmonary complications, its use remains limited [2,3,4]. This is explained by three main obstacles. First, controlling intra-operative bleeding using laparoscopic devices can be challenging and requires advanced technical skills [4]. Second, the surgeon cannot manually palpate the liver and thus cannot locate tumors and their resection margins easily, consequently creating a risk of inadequate oncological resection [5]. Third, laparoscopic ultrasonography (LUS), which is the only available tool for intra-operative tumor location, follows a steep learning curve to accurately visualize and interpret the subsurface anatomy [4, 5], especially in the posterior segments. In order to alleviate these obstacles, an increasing number of studies focused on using augmented reality (AR) during LLR are being conducted by several teams worldwide [6,7,8]. The objective of AR is to use preoperative imaging to overlay the internal anatomical structures onto the laparoscopic image (Fig. 1). This makes the patient’s organ virtually transparent and allows the surgeon to see the hidden subsurface anatomy such as vessels and tumors (Fig. 2) [9,10,11]. Previous work in AR guidance for LLR in monocular laparoscopy is mostly based on manually overlaying a rigid preoperative 3D model on the laparoscopic image [9,10,11]. This has a limited accuracy, keeping in mind that the liver is strongly deformed during surgery compared to its preoperative shape, due to intra-operative pneumoperitoneum, gravity, and mobilization [12,13,14].

Fig. 1
figure 1

A snapshot from CT segmentation (A) and the reconstructed 3D liver model with marked visible contours of the liver (B). The model shows the two tumors in green, the falciform ligament in dark blue and a ridge in light blue. (C) Example of manual marking of the anatomical landmarks on the laparoscopic image. The liver’s silhouette in yellow, the ridge in light blue, the falciform ligament in dark blue (Color figure online)

Full size image
Fig. 2
figure 2

Overlaid preoperative 3D liver model on a laparoscopy image with opaque tumors and transparent surface (A) or wireframe surface (B) visualization at the beginning of surgery

Full size image

Since a few years, our team has developed a software for AR guidance in monocular LLR called Hepataug. It uses a biomechanical deformable preoperative 3D liver model and so-called visual cues to capture the 3D shape of the liver from the laparoscopic image [6]. We have reported Hepataug’s technical details, use principle and potential interest in previous studies [15, 16], which have allowed us to continuously improve Hepataug. Importantly, we have assessed the accuracy of Hepataug on computer-simulated images [6], a 3D printed phantom [8] and an ex-vivo sheep liver [8] in previous pre-clinical tests. The average 3D error we found lies within a few millimeters, typically being of the order of 1–3 mm. These results were highly encouraging. The first step to take Hepataug to LLR procedures, which is our primary objective in this study, is to assess feasibility. Our secondary objective is to assess the potential benefit.

Methods

Inclusion criteria

Since July 2017, all the patients planned for LLR in the University Hospital of Clermont-Ferrand were included in this feasibility study. The patients were operated by a liver laparoscopic surgeon (EB or BLR) and only needed a preoperative CT scan or an MRI to be available and performed one month at most before surgery, and showing a liver tumor. The type, size, and number of tumors were not exclusion criteria, neither was the quality of the hepatic parenchyma. Liver resections were defined according to Couinaud’s classification and then divided in 4 categories, namely major hepatectomy, minor hepatectomy, segmentectomy, and tumorectomy. As this is a feasibility study, the indications for LLR were not modified by the use of AR. Signed consent was obtained, which included a clause of no modification of the surgery. The anonymous data collection was supported by an ethical approval with ID IRB00008526-2019-CE58 issued by CPP Sud-Est VI in Clermont-Ferrand, France.

Surgical technique

LLR was performed using three ports initially, with addition of others ports depending on the surgical procedure and operator preference. Laparoscopic ultrasonography was routinely used. Tissue dissection and hemostasis were performed using the ultrasonic dissector (CUSA®, Integra Lifesciences), bipolar forceps, and thermofusion. The major hepatic veins were divided using a vascular stapler. Intermittent Pringle maneuver as in open procedures was used in case of bleeding only. The resected specimen was placed in a plastic bag and removed through a suprapubic incision without muscle section. This incision was immediately closed and the abdomen reinsufflated to confirm hemostasis and the absence of bile leaks. Methylene blue or air injection through the cystic drain was not routinely performed. Abdominal drainage was only used if there was concern about intra-operative bile control or the adequacy of hemostasis. All intra-operative parameters, including blood loss with subsequent red cell transfusion, duration of surgery, Pringle maneuver and duration of the maneuver, were recorded. The patient was then discharged in postoperative care unit.

AR guidance with Hepataug

Hepataug's requires loading the patient's preoperative 3D liver model obtained from the preoperative CT or MRI and a single laparoscopic image obtained intra-operatively. The preoperative 3D liver model is reconstructed by segmenting the CT. It includes the liver’s shape and intraparenchymal landmarks, namely the tumors, the inferior vena cava and the hepatic veins (Fig. 1A and 1B). At the time of surgery, Hepataug runs in the operating room on a separate computer, which is placed next to the laparoscopy screen. Hepataug is run at the beginning of the surgical procedure, specifically at the time of exploration of the abdominal cavity. The laparoscope is used to visualize the liver for a few seconds, showing it as entirely as possible, and the images are recorded into the computer. The surgeon then selects a still image of the liver and the assistant marks the liver's visible contours (Fig. 1C). Finally, Hepataug automatically overlays the liver's deformable preoperative 3D model onto the laparoscopic image (Fig. 2). The accuracy of this registration was assessed visually, but also by comparison with LUS, notably for the deep limits of the tumors. LUS was used in all cases as usual. In cases for which LUS was not interpretable or not helpful due to intra-operative artifacts or other difficulties to locate the tumors, Hepataug was assessed by visual inspection of the preoperative CT or MRI. In these cases, Hepataug was run multiple times to assess its ability to predict the tumor location at the beginning of surgery and during liver transection.

Endpoints and data collection

The primary endpoint is the feasibility of using Hepataug in LLR. Specifically, the feasibility criterion is that Hepataug can be run smoothly in the operating theatre, without significant interruption of the surgical workflow. The feasibility was also assessed by collecting the conversion rate to open surgery, R0 resection (defined by a tumor free margin greater than 1 cm on the definitive histopathological analysis), blood loss, duration of surgery, pedicle clamping, perioperative transfusion, surgical events, postoperative complications, hospital stay, distant events, resection margins, and disease recurrence. The secondary endpoint is the assessment of Hepataug. The surgeon subjectively classified the correlation between LUS and Hepataug as poor, moderate, or good. A good correlation was stated when all the landmarks (edge of the liver, tumors, inferior vena cava) were consistent, a poor correlation stated when all the landmarks were inconsistent, and a moderate correlation stated when a mix of consistent and inconsistent landmarks was found. When LUS was not usable, Hepataug was assessed with a careful visual inspection of the preoperative CT or MRI.

Statistical analysis

Statistical analyses were descriptive. They were performed using Stata software, version 13 (StataCorp, College Station, US). Continuous data were presented as median and interquartile range.

Results

From July 2017 to March 2019, 17 patients were included in this study. The clinical data are summarized in Table 1. There were 8 women and 9 men, with a median age of 63 [IQR: 38–81]. The median BMI was 27.7 [IQR: 23–29]. The median tumor size of the biggest lesion measured on the preoperative imaging was 30 mm [IQR: 16–49] and the median cumulative tumor size was 30 mm [IQR: 25–50]. Among the 17 patients, 9 patients (53%) had liver disease or cirrhosis. Conversion to laparotomy occurred in 1 case (5%). Intermittent clamping was used in 15 procedures (88%). Median blood loss was 260 mL [IQR: 200–500]. The median hospital stay was 6 days [IQR: 5–8]. Postoperative severe complications that needed reintervention occurred in two cases (10%). These were a postoperative evisceration requiring abdominal wall closure and a biliary fistula requiring abdominal lavage and drainage. These complications were unrelated to the assessment of AR guidance feasibility, because therapeutic decisions were made independently of Hepataug. On the definitive histopathological analysis, all patients had R0 resection. In one case of benign tumor the transection plane was in contact with the lesion.

Table 1 Pre- and intra-operative data of included patients
Full size table

Regarding our primary objective of feasibility assessment, we did not encounter a signification disruption of the surgical workflow for any of the procedures by using Hepataug. This strongly suggests that using Hepataug, hence AR with a deformable model, in LLR is feasible.

Regarding our secondary objective of potential benefit assessment, we observed a good correlation between LUS and AR given by the surgeons in all 15 cases were LUS could be successful used, and the absence of a moderate or poor correlation. In the 2 remaining cases, Hepataug allowed us to see lesions that were not detected by LUS. The surgeon then performed a careful visual inspection of the preoperative CT or MRI next to the laparoscopic image. This allowed them to confirm and refine the tumor positions predicted by Hepataug for these 2 cases, whose final position assessment was hence rendered independent of AR assistance. We now describe these 2 cases in detail.

Case 1 The first patient for whom LUS was not predictive presented two metachronous colorectal liver metastases and was scheduled for laparoscopic wedge resection of these two lesions of 10 and 15 mm in segment 6. Because of their small size, their very lateral localization and the poor contrast with the non-tumoral parenchyma, the smallest lesion was not properly identified with LUS. We ran Hepataug a first time before starting the procedure. The predicted localization of the two lesions coincided with the improper LUS visualization and was confirmed by visual preoperative CT inspection. The resection line was then marked using electrocautery with 1 cm of free margins and the tumors were removed in-block. After the initial resection, LUS was used again to check the margins but was not contributive. We ran Hepataug a second time and confirmed the result from visual preoperative CT inspection. This required leaving the resected liver part temporarily in place. This crucially revealed that a margin on one side was not safe, with the transection plane in contact with the lesion (Fig. 3). Hence, the surgeon decided to perform a complementary resection to achieve a safest margin. The complementary resection of the inferior part of S6 was performed. On the histopathological analysis, the resection margins were greater than 1 cm after the complementary resection, thereby respecting the oncological rules. On the other hand, the margins of the initial resection did not exceed 1 mm. This case confirmed than running Hepataug twice during the procedure is feasible and demonstrates its potential interest in resection margin assessment.

Fig. 3
figure 3

Overlaid preoperative 3D liver model with opaque tumor visualization and transparent surface of the liver after the first resection. This shows that the free margins were not safe (Case 1)

Full size image

Case 2 The second patient for whom LUS was not predictive was scheduled for a double resection of two HCC lesions located in segment 5 and segment 7 in a cirrhotic liver. The first lesion was resected without difficulty. The second lesion, however, which measured 11 mm on the preoperative MRI, was not visible with LUS due to macro-nodular liver cirrhosis. Hepataug was run and predicted the tumor location, which was confirmed and refined by visual inspection of the preoperative MRI. This process was repeated 5 times during the procedure (Fig. 4). Histopathological analysis showed a complete resection of lesions with clear margins. This case confirmed than running Hepataug multiple times during the procedure is feasible and demonstrates its potential interest in resection margin assessment.

Fig. 4
figure 4

Overlaid preoperative 3D liver model with opaque tumor visualization and transparent surface of the liver (Case 2)

Full size image

Discussion

This case series reported the feasibility and the potential interest of using Hepataug to achieve AR with a deformable model in LLR to locate tumors. In this case series, AR guidance was feasible in all 17 patients. For 15 patients, LUS could give a reliable prediction of the tumor location, in good agreement with Hepataug’s prediction. For 2 patients, LUS could not give a reliable prediction of the tumor location. However, a careful visual inspection of the preoperative CT or MRI showed a good agreement with Hepataug’s prediction. This represents a first confirmation of the potential interest of Hepataug in surgical practice. The quantitative 3D accuracy of Hepataug cannot be assessed in real surgery because, by definition, one lacks a tool or method to infer the true location of subsurface organ structures to which augmented reality predictions could be compared with. Nonetheless, the accuracy of Hepataug has been assessed in previous pre-clinical studies on computer-simulated images, a 3D printed phantom and a sheep liver [6, 8]. We are currently working on a new and more realistic ex-vivo model using synthetic tumors and open ultrasonography to further these results.

Extra specialized equipment is not required for the use of Hepataug. It can be thus be easily used in the operative room. To obtain the final prediction of tumor location and AR guidance, the semi-autonomous registration time varied approximately between 1 to 3 min. In our experience, the first strength of AR is its ability to visualize internal parts regardless of the liver structure, irregularity of the transection plane, blood, and artifacts. AR provides a full 3D augmentation on the laparoscopy screen while LUS provides only local 2D cross-sectional images. AR is very interesting for cases of HCC on severe cirrhosis that can sometimes be unidentifiable by LUS due to macro-nodular lesions (as in case 2, Fig. 4). The assessment of free margins at the end of the surgery becomes possible and can be very interesting in the case of suspected invasion margins. AR also has the ability to easily locate and augment the subsurface structures difficult to access for LUS (posterior segments, vascular structures). Consequently, AR forms a powerful guidance tool for LLR. The cause of conversion to open surgery, which occurred in one case, was due to a difficult control of intra-operative bleeding and was not associated with the tumor location.

To the best of our knowledge, this study is the first case series on AR during laparoscopic liver surgery with a deformable model. As reported in Table 1, AR can potentially be interesting for various indications, from healthy to cirrhotic livers, and in a wide range of interventions from simple tumorectomy, to localize the lesion, to right or left hepatectomy, to check the lesion and choosing the transection plane. This study is thus highly encouraging regarding the feasibility and potential benefits of AR in LLR. However, the low number of patients and the fact that this study is not comparative does not allow us to conclude about efficacy.

Concerning case 1, the patient with colorectal liver metastases, Hepataug followed by visual preoperative CT inspection allowed us to locate tumors at the beginning of the operation and to define the transection plane on the surface area of the liver. However, as in open surgery, the transection plane may deviate during the procedure and free margins may decrease. For this reason, at the end of the first resection, the margins were lower than 1 cm. We showed that the problem can be handled by re-running the process of tumor localization several times during surgery. A better way to avoid this type of problems would be to have a system able to track the liver in real-time during surgery and achieve continuous AR. However, to date, there is no technical solution to achieve this, and we are actively researching the problem in our team. As reported in a previous study, liver registration in monocular laparoscopy is extremely difficult to solve [6, 8]. The main obstacles are factors contributing to the change of the organ’s representation intra-operatively compared to the preoperative imaging. These factors are cardiopulmonary motion, gas insufflation, gravity acting in different directions, and evolution of the disease. Furthermore, the monocular laparoscope does not provide the notion of depth as in a 3D camera, which increases the difficulty of registration. The liver is also only partially visible during surgery.

LUS is currently the only available intra-operative guidance tool. Surgeon experience is crucial in order to accurately localize the lesions, to check the resection margins before transection, to identify the important vascular structures, and to detect the presence of other lesions. On the other hand, the use of LUS is less ergonomic than open ultrasonography because the images are challenging to interpret due to various artifacts such as parenchymal bleeding, defects along the cutting edge, and difficulty to maintain the LUS probe in a correct axis along the transection plane. AR does not share these difficulties and is thus an interesting way to explore.

The literature in image-guided surgery is substantial and AR guidance in liver surgery has already been tested by several teams, including the IRCAD group in Strasbourg [12,13,14]. However, their preoperative model was designed for open surgery and overlaid onto the images manually, which can be inaccurate [17]. The strength of AR reported in their study is the ability to locate disappeared liver lesions, tumors which were visible in the original pretherapeutic CT scan but disappeared during the neoadjuvant treatment. Recently, Clements et al. have discussed an image-guided liver surgery system with deformation correction [18]. This procedure seems to be very interesting but their guidance system, designed for open surgery, requires the surgeon to swab the visible liver surface with an optically tracked stylus to reconstruct the visible surface in 3D, and overlays the preoperative deformable 3D model only onto the reconstructed 3D visible liver surface. The other methods proposed to solve liver registration automatically do not take account of gravity and pneumoperitoneum [9, 19]. There exist advanced methods which solve the problem but were designed to work in different conditions from monocular laparoscopy, because these need multiple intra-operative images, an intra-operative CT scan, multiple cameras, or a stereoscopic camera [20, 21]. In contrast, Hepataug is designed for monocular laparoscopic liver surgery, even if it could also be used in open liver surgery and in robotic surgery. It is a semi-automatic solution, which simply requires one to outline the liver’s contour on the laparoscopy image. Once this is done, the method has all the inputs it needs to produce a solution automatically in under a minute of computation.

This feasibility study concludes that AR with a deformable 3D model is feasible in LLR and has potential benefits in some cases, by its ability to locate tumors invisible to LUS. Comparative studies are needed to further assess the interest and efficacy of AR during LLR.