Introduction

One of the main current limitations of laparoscopy is the difficulty to accurately localize the target organ’s internal anatomy, owing to the absence of tactile feedback. This is a particularly important issue for the liver, which may contain malignant tumours to be precisely resected with an oncologic margin. Augmented Reality (AR) is a promising approach to overcome this limitation. The key idea is to overlay information extracted from a preoperative CT volume onto the laparoscopy images. These information may be the tumours and their oncologic margin but also the vascular structure. During the initial exploration phase of a surgery, AR allows the surgeon to perform resection planning. An example of augmented laparoscopic image is shown in Fig. 1. The laparoscopic image is overlaid with the projection onto the liver surface of the tumour’s boundary which is invisible to the laparoscope, along with a planned resection path following the oncologic margin. Compared to classical mental mapping approaches used in laparoscopy such as Ref. 12, AR systems like the proposed one cope with the deformation undergone by the liver from the preoperative to the intraoperative stages. Also, by directly overlaying the laparoscopic images with the registered preoperative model of the intraparenchymal structures, rather than mentally mapping them from a separate screen to the laparoscopic image, surgeons can be more confident regarding the real location of these structures.

A typical AR-based guidance system for laparoscopy is composed of two stages: (i) an initial registration stage during which the preoperative 3D model is aligned, or registered, to intraoperative laparoscopic images; (ii) an update stage during which the model is automatically registered to a new laparoscopic image by tracking visual cues. Current systems handle limited smooth changes at best, such as those induced by breathing, heart beating and manipulations with surgical instruments, but fail with irregular changes such as cutting or tearing. In this work we focus on the initial smooth deformable registration stage (i) which is a very challenging and currently highly researched problem. Its principle is illustrated in Fig. 2.

Figure 1
figure 1

Example of AR used to overlay a laparoscopic image with the projection of a tumour’s boundary (in red) with its oncologic margin (in green) onto the liver’s surface, as produced by our system. The subsurface tumour (in yellow) and major vessels (in blue) are also made visible. The planned resection path is marked in blue. Red dots are placed at every centimeter from the liver surface to the tumour. The resection mark follows the 1 centimeter oncologic margin advised in the literature for the treatment of colorectal cancer liver metastasis (CRLM) and hepatocellular carcinoma (HCC).

Full size image

The difficulty of the registration problem is two-fold. First, the liver is only partially visible in the laparoscopy image due to its large size and proximity to the laparoscope. Second, the liver deforms substantially between the preoperative volume and the laparoscopy image due to the pneumoperitoneum (the intraoperative CO\(_{2}\) gas insufflation) and its manipulation by the surgical instruments. We focus on regular laparoscopy, which in terms of computer vision is a single monocular pin-hole camera, and forms the standard in operating theatres. It is obvious that any system designed for monocular laparoscopy extends to stereo-laparoscopy.

Figure 2
figure 2

Registration of a preoperative 3D liver model with a laparoscopic image. The preoperative 3D model is extracted from CT and the camera represents the laparoscope. The preoperative 3D model comprises the liver volume whose external surface is in gray and parts of its inner anatomy, namely a subsurface tumour in yellow and vein in blue.

Full size image
Figure 3
figure 3

Registration results delivered by the state-of-the-art methods and the proposed one. (top left) The input laparoscopic image. (top right) Results from the automatic method11 based on visual cues (with liver parenchyma overlaying in gray, and contour constraints in yellow, blue and red). (bottom left) Results of the manual rigid registration method.15 (bottom right) Results of the proposed hybrid method, combining visual cues with a biomechanical model through cage-based tactile interaction. The cage’s control points (red dots) are used to edit the registration simultaneously with an automatic optimization procedure exploiting the visual cues.

Full size image

Currently, the most promising registration approaches share two main features. First, they solve the registration from the image contents only, without resorting to external hardware. Second, they use a preoperative 3D model consisting of the liver, tumours and vessels surface reconstructed by segmenting the preoperative volume. From these, the state-of-the-art registration methods are either manual15 or automatic.1,11,17 In Ref. 15, the preoperative 3D model is rigidly registered to the laparoscopy image by means of user interaction. In Refs. 1, 11, and 17, the preoperative model is deformed following a biomechanical model via an Iterative Closest Point (ICP)-like procedure to fit visual cues extracted from the laparoscopy image. These visual cues are anatomical landmarks including the falciform ligament and the inferior ridge, and the silhouette. The current manual and automatic approaches both present important shortcomings, illustrated in Fig. 3. In Ref. 15, the rigidity assumption is far too restrictive to accurately model the liver deformation. In Refs. 1, 11, and 17, the visual cues are sparse and do not convey enough information to unambiguously constrain registration. Though the reasons are different, this results in both cases in misregistration, impairing the reliability of AR.

We propose an hybrid registration approach. The key idea is that the manual and automatic approaches are highly complementary. Our hybrid approach extends and draws on the strengths of both by combining user interaction with visual cues and a biomechanical model. In other words, the rational is that both the machine and the user perception are valuable and should be taken into account via the visual cues and interaction respectively. In the presence of both user interaction and visual cues, our hybrid approach bundles all constraints in a single registration. In the absence of user interaction, it behaves similarly to the existing automatic approaches. In the absence of visual cues, it allows the user to edit the registration under guidance of the biomechanical model. This is a significant improvement compared to the existing manual approach as it allows the user to fully express their expertise in anatomy, prior experience and spatial understanding of the case at hand to the system. We have implemented this idea following the cage-based paradigm from the field of shape editing. The cage is a set of handle points enclosing the organ. Dragging these handle points interactively deforms the model. Shape editing is a widely studied problem. The cage-based paradigm is well-adapted to registration owing to its flexibility.

Concretely, we implemented our hybrid method with a Qt Graphical User Interface (GUI). Our system is entirely controllable by tactile interaction and may be used in a fast and intuitive manner. We compared our method named hybrid biomechanical (HB) quantitatively in fours ways against two previous methods,15 named manual rigid (MR), and11 named automatic biomechanical (AB). The first evaluation uses a silicon liver phantom faithfully reproducing the shape of a patient’s liver obtained from CT reconstruction. The phantom was deformed and we used Structure-from-Motion to reconstruct its ground-truth 3D shape. The registration was then tested for 20 views from 4 different deformation datasets of 5 views each. The registration error is defined as the average distance between vertices of the preoperative and ground-truth models. The registration error was evaluated for the visible and hidden parts. The second evaluation was performed over 7 images from 7 patients. The registrations were performed for every patient by 5 surgeons. Their manual interactions included the visual cues marking for AB and HB. The obtained registration results were used to evaluate and compare the inter-user registration variability, along with the 2D reprojection errors in the original view used for registration and a control view acquired from the laparoscope inserted in another optical trocar. The registration variability, defined as the root mean square of the standard deviation of the vertex positions, was evaluated over all the vertices and over the visible ones on the registered 3D model. The third evaluation consisted in measuring 2D reprojection errors measured between the set of occluding contour fragments of the liver visible in the laparoscopy image and the boundaries of the registered 3D model’s silhouette in both the original and control views. The fourth evaluation consisted in measuring registration errors on an ex vivo sheep liver. Three inner artificial tumours were injected into the liver. CT scans of the liver’s initial and deformed states were made to obtain the preoperative and groundtruth 3D models respectively. The registration error was evaluated for two laparoscopic views for MR, AB and HB for the three tumours.

Related Work

We review related work on biomechanical registration of a preoperative 3D liver model with laparoscopic images and on 3D shape editing.

Liver Preoperative Model Registration

This review is split in methods using the image contents only in monocular laparoscopy and methods using other modalities.

Monocular laparoscopy Methods1,11,17 process a single laparoscopy image with manually marked contour constraints representing the visual cues. More specifically, Refs. 11 and 17 rely on contours, namely the falciform ligament and inferior ridge, and the silhouette, whereas Ref. 1 relies solely on the silhouette. Method11 also uses a shading cue while Ref. 17 exploits environment priors modeling the effect of the pneumoperitoneum and gravity. Exploiting these environment priors remains difficult in vivo because of the unknown boundary conditions involving the viscera. These methods are highly desirable as being compatible with standard laparoscopy. One of their main limitations is that occluded parts are still poorly registered which has a direct impact on the location of registered tumours and vessels.

Non-monocular laparoscopy Methods11,12,13,14,15,16,17,18,19,20,21 use a stereo-laparoscope to reconstruct the visible surface of the intraoperative liver shape. In Ref. 10, the liver’s 3D contour boundaries are automatically detected on the visible surface and used to constrain registration. Method18 extracts 3D features on the preoperative and intraoperative surface meshes and robustly finds correspondences using the feature descriptors and locations. The two shapes are then aligned through a rigid registration. Method21 reconstructs surface patches of the intraoperative liver shape. The stereo-laparoscope is tracked using an optical tracking system. This allows one to localize the patches in world coordinates and use them to constrain registration. Method8 uses a tracked stylus to let the user enter landmarks on the liver surface. Because the pose of the landmarks is known, they directly serve as registration constraints. Method16 uses intraoperative CT scans to constrain registration. Finally, method7 registers an intraoperative CT scan to the laparoscope by imaging the laparoscope’s distal end itself within the CT volume and combining this with shading. These methods share a dependency on non standard laparoscopy or special hardware to solve registration. Nonetheless, with the exception of Ref. 16, they do not address the problem of registering the liver’s hidden parts, strongly limiting their usage for AR.

Interactive Shape Editing

Shape editing refers to the change of a model’s surface through a set of handles either part of or connected to it. Existing approaches can be divided in four main categories, depending on how such handles are distributed: point-based,5 curve-based,4 surface-based9 and cage-based deformations.20 In a point-based approach, the user provides a set of point displacements, each comprising a point along with its intended motion and region of influence. The way points are distributed does not depend on the shape of the model, but more on the user’s preference. When these points are moved, the object is then warped to match the displacement constraints.5 In curve-based approaches, the deformations are controlled by one or more curves. The control points are distributed to form a line that the user curves. The deforming object is distorted to map from the source to the destination curves.4 The surface-based approaches consist in deforming the object when a surface patch is modified by translating a set of control points. The control points are directly located on the surface of the model. One of the main difficulties is to find a way to attach sample points on the object to the deforming patch.9 The cage-based approaches use a cage that encloses the object. This cage can have a fixed shape such as a cuboid,20 or can be adapted to the shape of the object to be deformed.19 The shape of this cage is altered by repositioning control points. The resulting cage distortion is then transferred to the object.

Materials and Methods

Hybrid Registration

We first describe the principle and pipeline of our method. We then describe our implementation of the biomechanical model and the visual cues constraints. We finally show how these integrate with cage-based user interaction.

Principle and Pipeline

Our hybrid registration method takes as input a preoperative 3D model and a single laparoscopic image. Its principle is to combine a biomechanical model and the manual and automatic registration approaches. These respectively use user interaction and visual cues extracted from the image to solve for registration. Our method thus rests on three sets of constraints. The first two are borrowed from Ref. 11. These are a biomechanical model based on the Neo-Hookean elastic model and the use of the falciform ligament and inferior ridge as curve correspondences, and the silhouette. The third set of constraints are the cage-based constraints to model user interactions. Concretely, the preoperative 3D model is represented by a tetrahedral mesh and optimization follows the principle of position-based dynamics.6

The pipeline of our method is illustrated in Fig. 4. It has 7 main steps. The first two steps are similar to Ref. 11: in step (1), the user marks the falciform ligament, inferior ridge and silhouette on the laparoscopy image and in step (2), the user marks the corresponding contours on the preoperative 3D model. In step (3), the system generates a cage enclosing the preoperative 3D model, to be used for user interaction at step (6). In step (4), the cage’s control points and the preoperative 3D model are co-tetrahedrised in order to obtain a single tetrahedral model. In step (5), an initial registration is computed using only the visual cues, following an automatic method.11 This initial registration is required to initiate interactive registration. In step (6), the user interactively edits the registration by moving the cage’s control points. The registration is updated in real-time to provide the user with live feedback. Importantly at this step, both the cage’s control points and the visual cues are used to update the registration. Finally, once the user is satisfied with the registration, step (7) augments the laparoscopic image with hidden anatomical elements transferred from the preoperative 3D model.

Figure 4
figure 4

Pipeline of the proposed hybrid 3D to 2D deformable liver registration method. The liver surface mesh is overlaid in gray, its subsurface tumour in yellow and vein in blue. The contours associated to the silhouette, the falciform ligament and the ridge are marked in yellow, blue and red, respectively. The cage is rendered in blue wireframe and its associated control points with red dots.

Full size image

Biomechanical Model and Visual Cues Constraints

The biomechanical model is created by augmenting the preoperative 3D model with the isotropic Neo-Hookean elastic model.6 This non-linear hyperelastic model works well for registration in laparoscopy, which involves moderate deformations, under the following three conditions: after the liver is freed from the falciform and round ligaments (which is always done at the start of surgery for accessibility purposes), when there is no strong external forces from the tools, and before any resection takes place. The associated mechanical parameters are set to generic values measured for the liver, namely the Young’s modulus to \(E=60,000\) Pa and Poisson’s ratio to \(v=0.49\).17 The contour constraints rely on anatomical landmarks which are the ridge, the falciform ligament, and the silhouette contours to constrain the deformation. The ridge contour is located at the bottom of the liver, it is almost always visible and it has a very distinctive profile. The falciform ligament attaches the liver to the abdominal wall. It is located in the separation zone between the left and right lobes. It is cut in the early stage of surgery to let the liver move freely. Its location is then made clearly visible on the liver external surface. These two sets of contour fragments are stationary. Their correspondence with vertices in the preoperative 3D model remains fixed for the entire registration procedure. The last set of contour is the liver silhouette, imposing the liver model to not deform beyond those boundaries. Unlike for the ridge and the falciform ligament contours, the silhouette contours are not stationary and the associated set of constraints must be updated during the registration procedure. Because a silhouette contour corresponds to the upper convex diaphragmatic surface of the liver, which is a very smooth region, the silhouette curve can slide on the surface as the optimization progresses. In contrast, because of the very well defined curvature profile of the ridge, and the narrowness of the falciform ligament landmark, we can make use of the same set of vertices in the 3D preoperative model as correspondences for their 2D counterparts, and thus prevent the model to freely ‘slide’ on the surface during optimisation. All these contour constraints are introduced in the optimization algorithm using an ICP technique. We do not explicitly model physical factors like diaphragm pressure, pneumoperitoneum or pre-stretching of the liver as boundary constraints for the registration process, as these are not measurable, both preoperatively and intraoperatively. Instead, these effects are handled by the interactions of the surgeon with the preoperative 3D models through the surrounding cage.

Cage-Based User Interaction

An intuitive and easy-to-use interface allowing the user to edit the liver’s shape in a way that respects its properties and the visual constraints must be proposed. This is achieved through the use of a cage. This has a good trade-off between editing flexibility, namely the possibility to edit at an appropriate spatial frequency, and user friendliness. The cage is represented by a mesh composed of a very limited number of control points. These are obtained following the cage initialization procedure defined in Ref. 19 so that the cage encloses the input preoperative 3D model. Once the cage is generated, it is linked to the preoperative 3D model through a Delaunay tetrahedralization applied on all the vertices, namely the cage, the liver and its inner structures vertices. During the registration, they are all optimized using the same material model. An example of a generated cage is shown in Fig. 3.

Making the constraints derived from the cage movable during the optimization procedure is not trivial. We propose to embed once and for all the cage’s control points into the volumetric model of tetrahedral topology built from the preoperative 3D liver model. Another possible solution would consist in creating a cage according to the model deformation at every iteration, which would however significantly harm the software usability. When a cage’s control point is moved during the optimization, all the vertices belonging to the liver model adjacent to it are moved accordingly. This allows the user to handle a set of vertices simultaneously over a model region. These deformations are compensated by the constraints described in “Biomechanical Model and Visual Cues Constraints” section at the same time. During optimization, a single iteration of the contour-based optimization is run for every change in position of the cage vertices in order to increase the responsiveness of the deformation.

Tactile Graphical User Interface

The proposed GUI is shown in Fig. 5. It is divided in four sections. First, the visualization area in which the laparoscopic image and the preoperative 3D model are shown. The user can position the preoperative 3D model and mark the contours using tactile gestures or the keyboard and mouse. Second, the left toolbar, which is used to either import or export the laparoscopic image and the preoperative 3D model. Third, the right toolbar, which is used to modify the appearance of the preoperative 3D model, to mark the visual cues and to launch registration. Fourth, the bottom toolbar, which controls the size of the visualization area, lets the user activate the cage-based editing mode, and implements miscellaneous other functionalities.

Figure 5
figure 5

Proposed tactile Graphical User Interface for our hybrid method.

Full size image

Registration begins with a user click on to set the laparoscope parameters obtained from a prior calibration procedure. The button is then used to load the laparoscopic image and the button to load the preoperative 3D model. The visual cues are marked both in the laparoscopic image and the preoperative 3D model with the help of the controls located in the right toolbar. A rigid registration is done automatically so that the preoperative model fits the liver approximately in the laparoscopic image. The nested cage is generated by clicking on . The user then proceeds to match the contours by clicking on . The preoperative 3D model can then be translated and rotated so that they approximately fit the image. Automatic contour-based deformation is then launched by clicking on . Once it completes, the user can proceed to edit the registration using the cage by clicking on the button . The vertices of the cage may be dragged while the system displays the registration combining the visual cues and the cage in real-time.

Our AR software is setup on a PC computer running Linux. In the OR, this computer is connected via a capture card to the laparoscopy column in order to capture the laparoscopic video stream. The computer is located close to the other screens so that the surgeon has a direct view of the augmentation (see Fig. 6). It is equipped with a tactile screen, which can be directly used by the surgeon.

Figure 6
figure 6

Usage of our AR guidance system (left computer) in the operating room to perform a laparoscopic tumour resection.

Full size image

Results

This section is divided in three parts. In the first part, results of an accuracy evaluation of the registration on a liver phantom are reported. In the second part, the registration variability and the reprojection errors in control views are evaluated on in vivo liver data from the hepatobiliary and pancreatic surgery department of the CHU Estaing hospital in Clermont-Ferrand, France. In the third part, the registration accuracy is evaluated with respect to inner artificial tumours injected into an ex vivo sheep liver. We compared our method HB quantitatively in two ways against two previous methods, MR15 and AB.11

We recall that for pathologies such as colorectal cancer liver metastasis (CRLM) and hepatocellular carcinoma (HCC), a resection margin of 1 cm should be considered if possible.13,22 Thus, we consider a registration error nearby the tumour of a centimeter or lower as successful.

Accuracy Evaluation on a Liver Phantom

The accuracy of the proposed registration method is evaluated using a 3D printed liver phantom made of silicon (Fig. 7a) aiming to simulate the bio-mechanical properties of a real liver. The liver phantom was built as follows. The preoperative 3D model was segmented from CT data of a real patient. A mold of this 3D liver was generated and 3D-printed. The mold was finally filled with silicon (Fig. 7). We used an Ecoflex 00-20 silicon material made by Smooth-On Inc. which has a Young elastic modulus of 60 kPA,2 very close to the 50–60 kPa of a human liver.

Figure 7
figure 7

(a) Segmentation of the liver in a patient CT image. (b) Preoperative 3D liver model obtained from the segmentation and used to 3D print a model for (c) the liver phantom used for the proposed accuracy evaluation of the registration methods.

Full size image

The principle of this experiment is as follows. The liver phantom is deformed and its shape reconstructed using the Structure-from-Motion software Agisoft Photoscan,3 as shown in Fig. 9. Then, we take N views out of those used to reconstruct the phantom’s shape as input images for the registration procedure. The CAD model from which the phantom has been printed is used as the input preoperative 3D model and is registered following the proposed registration method (Fig. 8).

Figure 8
figure 8

Registration of preoperative 3D model on phantom data with HB method.

Full size image
Figure 9
figure 9

The deformations applied to the 3D printed phantom used in our first set of experiments. Top: deformed phantom. Bottom: 3D model, reconstructed with the Agisoft Photoscan software, used as ground-truth in our experiments.

Full size image

This experiment is performed for \(M=4\) phantom deformations, shown in Fig. 9, and \(N=5\) different views per deformation. The registration error, defined as the average distance between vertices of the registered preoperative and ground-truth models, is reported in Table 1. As we compared the distances between all the vertices and not only the ones involved in the registration, it can be considered a measurement of a target registration error (TRE).14 We report the errors for two sets of vertices: those associated to the entire registered model and those restricted to its visible part, namely the anterior part, corresponding to a usual laparoscopy image.

Table 1 Registration errors with respect to ground truth for the phantom experiment.
Full size table

HB shows the lowest registration error. The error of MR is noticeably higher as the method does not deal with the phantom deformations. The registration error of AB is overall lower than MR’s. It shows that the visual cues in AB well constrain the biomechanical model. HB shows lower errors than AB as misaligned parts can be corrected while preserving the visual cues and biomechanical constraints. The standard deviations are the lowest for HB which shows that the method also provides the most stable results. In some cases, such as for example the registrations AB performed on dataset 1 or HB on dataset 4, the average error over the entire liver is lower than the error over its visible part, which reveals lower registration errors on hidden parts. To better illustrate this, error distributions over the entire liver’s vertices are shown in Fig. 10.

Figure 10
figure 10

Error distribution over the registered phantom models using (a) MR, (b) AB and (c) HB. The colors range from blue which corresponds to the lowest registration errors to red which corresponds to the highest ones. Distances are in millimeters. The visible parts correspond to the areas limited by the purple curve. Top: cases associated to a registration error higher on the visible parts than on the hidden parts. Bottom: cases associated to a registration error higher on the hidden parts than on the visible parts.

Full size image

The registration accuracy is also evaluated with varying visibility of the liver phantom. A decreasing Field of View (FoV) was simulated by adding a circular black border to the images. The registration were performed for a FoV of 100, 70 and 50%. One image per dataset were used to perform MR, AB and HB registrations. The results are reported in Table 2. Some registration results are illustrated in Fig. 11.

Figure 11
figure 11

Registration results for 3 different FoV on the first dataset: a FoV of (a) 100%, (b) 70% and (c) 50%. The top images correspond to registrations using MR, the middle images to registrations using AB, and the bottom images to registrations using HB. The circles represent the FoV applied in each case.

Full size image
Table 2 Registration errors (in mm) for decreasing FoV using MR, AB and HB methods.
Full size table

The registration accuracy is also assessed for AB with a varying number of tetrahedrons composing the biomechanical model. Three preoperative 3D models were created, comprising tetrahedrons obtained by the triangulation applied on 8000, 4000 and 2000 vertices respectively. AB was run on one image per dataset. The registration error is reported in Table 3.

Table 3 Registration errors for the AB method using preoperative 3D models with varying number of vertices.
Full size table

Variability and Control View on In Vivo Liver Data

A high variability of the registration results obtained from different operators is a sign of unreliability of the registration solutions. High reprojection errors in control views reveal a bad registration. We propose to assess both registration variability and reprojection errors in control views on real patient’s data (Fig. 12).

Variability for MR, AB and HB Registrations

We asked 5 surgeons to perform MR, AB and HB registrations on seven different patient datasets. Before performing registration, the surgeons were also provided with short videos acquired during the surgery’s exploratory stage. The laparoscope was inserted in different trocars to let the surgeons have a wider scene perception. Table 4 reports the average of the vertex-to-vertex root-mean-square deviation (RMSD) over the surgeons for MR, AB, and HB registrations. For a patient, the RMSD measures how different the registered shapes are between surgeons. It differs from the standard deviations reported in Table 1 which correspond to the deviations of the registration errors computed from ground truths.

Table 4 Registration variability (in mm) over the surgeons on 7 in vivo datasets.
Full size table

The average variability for all the patients is of 9.1 mm for MR, 13.2 mm for AB, and 11.1 mm for HB. One of the key results is that, while HB offers a higher flexibility on the model deformation than for AB, the overall registration variability remains lower. MR shows the lowest variability because it has very little flexibility.

Control View

From the registrations made by the surgeons on the in vivo data, we selected 6 patients for which we had views of the liver acquired from a different optical trocar. We measured the 2D reprojection errors as the distance from occluding contours manually extracted from the laparoscopic images to the boundaries of the registered model’s reprojections. It was performed for both the reference and additional views. Tables 5 and 6 report the 2D reprojection errors in pixels for the reference and the additional views respectively. Table 7 reports the average of the reprojection errors for both views.

Table 5 Reprojection error (in pixels) of the in vivo patient data in the reference view.
Full size table
Table 6 Reprojection error (in pixels) of the in vivo patient data in the control view.
Full size table
Table 7 Reprojection error average (in pixels) of the in vivo patient data from both reference and control views.
Full size table

The average reprojection errors are much higher for MR than for AB and HB, while HB has the lowest values. The rigid model in MR cannot be correctly aligned to fit the imaged liver.

Registration Time

The total setup time is 05:56 (5 min and 56 s) on average in our experiments. This time can be split between time requiring the surgeon’s attention (understanding the scene, marking the landmarks and performing HB registration), which is 04:05, with standard deviation 00:38, and time not requiring the surgeon attention (for the software to initialize the system and compute AB registration), which is 01:05 on average. It is worth noting that, once the surgeon has understood the scene and made the first registration, subsequent registrations on the same patient will take less time.

Figure 12
figure 12

Patient images used for control view evaluation, along with two examples of augmented images and their reprojection errors after HB registration. Laparoscopic images have Full HD resolution (1920 \(\times\) 1080 pixels). The liver surface mesh is rendered in gray, their subsurface tumours in yellow and veins in blue.

Full size image

Accuracy Evaluation on Ex Vivo Sheep Liver

We assess the accuracy of our method with respect to inner tumours, by means of an ex vivo sheep liver which was injected with alginate to create three artificial tumours. Two CT scans of the liver were made. The first one was performed to build the preoperative 3D model to register. The liver was then deformed. The second CT scan was performed on the deformed liver together with a Structure-from-Motion based 3D reconstruction to obtain a registration ground truth (see Fig. 13). The registrations were made on two laparoscopic views of the deformed liver using MR, AB and HB for two degrees of visual cues visibility: low and regular. This emulates the possible occlusions from fat and the surrounding organs. The distances between the three registered tumours and their respective ground-truths were then measured. The results are reported in Table 8.

Figure 13
figure 13

(a) SfM-reconstructed model of the deformed liver. (b) CT-reconstructed models of the deformed liver and tumours. (c) SfM and CT models aligned using ABSOR.

Full size image
Table 8 Registration errors for three synthetic inner tumours on an ex vivo sheep liver using (a) low visibility and (b) regular visibility of the visual cues.
Full size table

We observe that method HB outperforms for all three tumours and both visibility levels. Methods MR and AB compete for the second best performance, depending on the tumour, though MR is overall slightly better.

Discussion

The registration errors obtained from our method are very promising. The user can expect a similar or lower range of error nearby the tumour area, an error which is below the 1 cm oncologic margin advised in the literature for tumour resection in laparoscopic hepatectomy. The low variability obtained from our method suggests that surgeons have a similar interpretation of the scene and were provided with an appropriate tool to edit the model shape accordingly. The lowest variability shown by MR can be explained by the limited control on the model compared to AB and HB, namely restricted to the model’s rigid pose. The time spent by surgeons to perform registration represents a very small portion of the total surgery time. Nonetheless, automating the detection of the landmarks could drastically decrease the manual interaction required from the surgeon, reducing the total registration time and thus improving usability. The problem of landmark detection in the laparoscopic image could be tackled within the framework of deep neural networks. However, it is a difficult problem which to date remains open. Contrarily to organ detection and segmentation, for which recent techniques show compelling results, landmark detection would require the machine to detect curves (which are more difficult to represent than regions in a deep neural network) and to classify them in a type related to their semantics (lying on or off the liver) and geometric properties (being part of the silhouette, for instance). This problem is still open in the computer vision and medical image processing literature.

The amount of visible liver also plays an important role in the registration, as shown in Table 2. The lack of visibility affects greatly MR, while AB and HB have better and consistent errors regardless the FoV size. This indicates that in such cases both AB and HB are able to recover the shape of the hidden parts successfully. In general, we see an increase in the registration error for a higher number of vertices/tetrahedrons in the preoperative 3D model, as seen in Table 3. Nevertheless this does not always hold for each individual dataset, which means that factors such as the viewpoint and the liver shape play a more important role in the registration than the number of tetrahedrons in the preoperative 3D model. Preliminary results on the ex vivo experiments show that our method is able to accurately recover the location of the inner tumours for a varying visibility degree of visual cues, even if they are far from any visual constraint and regardless the viewpoint used for registration, as shown in Table 8. The two levels of visibility bring an interesting observation: the stronger the visibility, the smaller the differences between the methods. Specifically, we observe that HB brings a substantial improvement when visibility decreases. This is a sensible result, because when visibility decreases, the added value of the surgeon expertise expressed by their interactions increases, maintaining the performance, while MR and AB may only worsen.

The registration performance remains correlated to the technical difficulties inherent to laparoscopic surgery, such as a reduced field of view and limited viewpoint range, which may substantially vary with patient anatomy. For example, the registration of a laparoscopic image where the liver is entirely visible and whose anatomical landmarks can be accurately localised (such as patient #5 in Table 5) is more accurate than one performed on an image where the liver is a partly visible and whose landmarks localisation is ambiguous (such as patient #3 in Table 5).

Our approach works on static laparoscopic images, which represent weak inputs, but nonetheless captures the effects of respiration, diaphragm interactions and pneumoperitoneum via the extracted visual cues. In other words, the visual cues inherently represent these complex constraints, which are not capturable otherwise in the routine surgical context of the problem at hand. The strength of our approach is to complement these visual cues which are also weak constraints, by surgeon interactions. This allows our system to take advantage of the observable landmarks from the input laparoscopic image (via the visual cues) and of the surgeon’s expertise and understanding of the intraoperative scene (via their interactions). Our results confirm that combining a biomechanical model constrained by visual cues and manual interactions is very fruitful. As future work, the registration software will be modified to let the surgeon choose the number of control points in the generated cage, according to the complexity of the liver’s shape. The influence of using a more advanced biomechanical model on the performance should also be evaluated. Further clinical tests have to be made in order to validate our method, notably regarding the location of inner structures after registration on human cases. If such tests confirm an overall registration error lower than 1 cm, then the proposed method will give surgeons a reliable basis to guide resection.