Keywords

1 Introduction

Minimally-invasive procedures for stenting or treatment of aneurysms use guide-wires [1, 2] to provide easier access through complex vascular structures. With these tools, and a combination of both fluoroscopy and CT-images as visual guidance, a surgeon navigates a catheter to difficult-to-reach anatomical sites such as side branches of the aorta or pulmonary arteries. However, regular use of CT or x-ray acquisition exposes clinicians over time to accumulated high doses of radiation. Upcoming robotically driven guidewires [3] have the potential to significantly reduce this exposure but require complex preoperative planning.

An efficient preoperative pipeline that transfers the burden of segmentation and trajectory planning to automated algorithms while keeping the surgeon in control of crucial parts is vital for the success of these new robotic approaches.

We consider two example anatomies for endovascular procedures, aorta and pulmonary arteries (Fig. 1), to evaluate a preoperative pipeline including segmentation of risk structures and nonlinear trajectory planning. To the best of our knowledge, no complete pipeline of preoperative planning for endovascular procedures has been proposed so far. Azizi et al. [4] exploited centerline extraction to search for collision-free piecewise linear trajectories serving as a guidance for tools. They evaluated their segmentation on five cases for arteries and a 3D model of a porcine portal artery. Chi et al. introduced reinforcement learning to further optimize paths on centerline extraction [3]. Their evaluation on three aortic arch phantoms considered navigation along trajectories only. Most recently, Fauser et al. [5] investigated sequential convex optimization for trajectory optimization in guidewire procedures. Their method was evaluated on the SegThor data set [6] but uses ground truth segmentation as a basis for computation.

In this paper, we investigate a full preoperative pipeline including both segmentation and trajectory planning. We identify three key challenges for complete evaluation:

  1. 1.

    Guaranteeing coherent shapes after segmentation, i.e. without fragmentation or isolated regions, because these interfere with collision detection.

  2. 2.

    Interactive definition of the motion planning problem. This includes placement of start and goal regions. But also correction of surface meshes, if neighboring labels (e.g. right ventricle and atrium) result in boundaries between structures where naturally there are transitions.

  3. 3.

    Providing a clinically optimal solution if motion planning algorithms do not guarantee optimality.

To address the first issue, we adapt the shape-regularized U-Net approach of [7, 8]: In data-scarce environments with only few annotated training images, deep learning architectures like U-Net [9] fail to guarantee coherent boundaries due to pixelwise prediction. Active Shape Models (ASM) [10] on the other hand intrinsically provide strong topological assumptions in terms of shapes but need proper initialization. A combination of both, where U-Net initializes ASMs [8], provides the best of both worlds: high accuracy and anatomically realistic shapes. For the second issue, we rely on the 3D environment provided by the resulting surface models. Here, a surgeon can efficiently define start and goal regions for trajectory planning as well as cutting holes into the existing surface meshes using adequate interaction. We tackle the last issue by using Bi-directional Rapidly-exploring Random Trees [11] with Bézier-Splines as steering functions [12] to compute multiple feasible trajectories. An iteration of sequential convex optimization [5, 13] improves clearance to obstacles for these paths. We then rely on the surgeon to identify the clinically optimal solution.

In our experiments we conduct a functional evaluation of the pipeline combining metrics on downstream tasks (trajectory planning) along with Dice and Hausdorff distances as segmentation metrics. Our results show that with shape regularized obstacles successful preoperative planning is possible.

Fig. 1.
figure 1

A trajectory through left: the aorta to the heart for a SegThor sample and right: the right atrium and ventricle to the pulmonary arteries for a MMWHS sample.

Full size image

2 Materials and Methods

Data: We evaluate our proposed framework on two publicly available data sets. First, using the 40 3D-CT thorax scans of this year’s SegTHOR challenge [6] with corresponding label images identifying heart, aorta, esophagus and trachea. Secondly, using partial data of 2017’s MMWHS challenge [14], specifically the two CT training sets offering a total of 20 3D-CT scans with corresponding label images discriminating between left ventricle, atrium and myocardium, right ventricle and atrium as well as ascending aorta and pulmonary artery (Fig. 1).

General Procedure: Both anatomies offer the possibility of evaluating the preoperative pipeline shown in Fig. 2: Based on a CT scan, a segmentation algorithm automatically extracts organs at risk while guaranteeing realistic shaped boundaries. In the resulting 3D anatomy the surgeon defines start and target states for the designated procedure. A motion planning algorithm then computes feasible trajectories for the instrument which are optimized regarding clearance to obstacles. This procedure automates laborious tasks, while at the same time giving the surgeon control over crucial parts of the pipeline.

Segmentation: Similar to [7], we rely on Deep Learning to properly initialize Active Shape Models (ASM).

U-Net: Taking a slice-by-slice approach, we first predict axial, sagittal and coronal slices using three individual 2D U-Nets \(\mathcal {U}_A, \mathcal {U}_S, \mathcal {U}_C\), respectively, and apply majority voting on these three results to receive an initial 3D segmentation image \(I^\mathcal {U}\). For trajectory planning, however, we need surface meshes of the detected structures’ boundaries. Using the Marching Cubes algorithm on \(I^\mathcal {U}\), surface meshes \(M^\mathcal {U}_i, 0\le i\le N,\) for each of the N organs are extracted (\(N=4\) for SegThor, \(N=7\) for MMWHS). All three U-Nets consist of 5 levels, each implementing one separable \(3\times 3\) convolution layer followed by ReLU activation and doubling the number of feature channels starting with 16 channels. A reverted sequence of 5 levels combining up-convolutions and skip connections finishes with a soft-max activation for a final segmentation output. We use RMSProp as optimizer with a learning rate of 0.0001 and train for 30 epochs.

ASM: For each of the risk structures we create a statistical shape model (SSM) \(\mathcal {S}_i, 0\le i\le N\). For correspondence search, we extract ground truth meshes \(M^{GT}_i\) using first Marching Cubes on label images and then coarsening these shapes to 5000 landmarks [15]. For initialization of an individual active shape model (ASM), we nonrigidly register the mean shape of its SSM to U-Net’s mesh result \(M^\mathcal {U}_i\). In particular, we use a Probabilistic ASM (PASM) which iteratively adapts its shape using an energy function that weighs between the projection into shape space and image information [16]. This results in a final segmentation mask \(I^\mathcal {P}\) and corresponding meshes \(M^\mathcal {P}_i, 0\le i\le N\).

Fig. 2.
figure 2

Preoperative pipeline: a CT scan serves as input for 2D U-Nets to predict an initial segmentation (A). Probabilistic ASMs regularize the shapes of fragmented structures (B). A surgeon then interactively defines start and goal states (pink circles) and creates openings (yellow circles) (C). Bi-RRTs find feasible trajectories (D, search graphs in pink & cyan). Computed paths can be locally optimized using SCO (E). (Color figure online)

Full size image

Trajectory Planning: We rely on the surgeon to interactively define start and goal for the guidewire. We use Bi-directional Rapidly-exploring Trees (Bi-RRT) to find initial solutions and sequential convex optimization (SCO) for enhancement of clearance to risk structures.

Interactive Setup: Given the 3D environment of risk structures, a surgeon initializes the motion planning problem. For guidewire planning in endovascular procedures, we specifically consider

  • a start state \(q_S\in C = \mathbb {R}^3\times {\text {SO(2)}}\) to enforce strict position and direction,

  • a goal state \(q_G\in C\) to enforce the same at the target,

  • an upper curvature constraint \(\kappa _{max}\ge 0\), representing the maximum bending capability of the guidewire,

  • a safety distance \(d_{min}\), combining the guidewires size and a safety margin to account for navigation and segmentation errors. We model the guidewire’s tip as a ball in \(\mathbb {R}^3\) with radius \(r_g>0\) and enforce a safety margin \(d_m>0\) from obstacles, resulting in \(d_{min} = r_g + d_m\).

This step requires interactive placement of a suitable start configuration \(q_S\) for the instrument as well as a specific goal state \(q_G\). Moreover, it includes the creation of transitions between neighboring labels: at the tricuspid valve between right atrium and ventricle, at the pulmonary valve between right ventricle and pulmonary artery and finally for SegThor at the aortic valve between aorta and left ventricle.

Motion Planning: We use a Bi-RRT with Bézier Splines as steering function [12] to search for collision-free trajectories. Computed paths interpolate between \(q_S\) and \(q_G\) while satisfying both constraints on distance and curvature. First, we compute multiple solutions in a fixed time interval \(T_{max}>0\). Finally we perform SCO on each of these solutions to optimize with respect to a cost function weighing between clearance to obstacles and trajectory length [5].

3 Experiments

Source Code: C++ code of methods and experiments is publicly available on https://github.com/MECLabTUDA for the benefit of the research community.

Segmentation: We divided the two data sets (MMWHS, SegThor) into two subsets (first & second half) for twofold cross validation. U-Nets and PASMs were trained on one subset. Prediction and planning was performed on the other. For the two outputs \(I^\mathcal {U}\) and \(I^\mathcal {P}\) we computed Dice scores and Hausdorff distances.

Planning: We used all three sets \(M^{GT}, M^\mathcal {U}, M^\mathcal {P}\) of surfaces meshes. First, the interactive setup of the preoperative pipeline (Step (C) in Fig. 2) was performed manually based on ground truth structures \(M^{GT}\). For SegThor, start and goal states \(q_I,q_G\) were placed within the lower part of the descending and inferior to the ascending aorta (Fig. 1). For MMWHS, \(q_I,q_G\) were placed within the right atrium and the pulmonary artery. After setting up the motion planning definition, we computed access paths using the Bi-RRT three times in a row: Once using obstacles based on \(M^{GT}\), once on \(M^\mathcal {U}\) and once on \(M^\mathcal {P}\), resulting in three sets of trajectories \(T^{GT}\), \(T^\mathcal {U}\), \(T^\mathcal {P}\). We then measured the success rate for planning on both U-Net and PASM results, i.e. the percentage of data sets where at least one path was found for \(T^{GT}\), \(T^\mathcal {U}\) and \(T^\mathcal {P}\). We then computed the distances to risk structures for \(T^{GT}\), \(T^\mathcal {U}\), \(T^\mathcal {P}\) using only shapes \(M^{GT}\) as obstacles. This resulted in computing the true distance to risk structures when planning on U-Net and PASM, respectively. We then measured the mean distance of the minimum distances along each trajectory as well as the failure rate, i.e. the number of data sets were a path computed on segmentation results was actually below the critical safety distance \(d_{min}\).

Fig. 3.
figure 3

Qualitative results on a SegThor sample (ground truth, U-Net & ours). Contrary to U-Net, our shape regularized approach provided shapes for feasible planning.

Full size image

4 Results

SegTHOR: A functional evaluation of a preoperative pipeline evaluates both segmentation and planning metrics. Table 1 shows Dice and Hausdorff distances (HD) for esophagus, heart, trachea and aorta. While dice is comparable, our approach achieves for most anatomies significant improvement on Hausdorff distance. The qualitative example in Fig. 3 shows that a combination of fragmentation and isolated regions is often responsible for bad quality in the U-Net approach whereas our shape regularized solution provides realistic and accurate segmentation for planning. A downstream analysis on trajectory planning evaluates the overall quality and usefulness of the segmentation results by adding metrics on motion planning. Table 2 shows the success rate of the motion planning algorithm from Sect. 2. The fragmented structures from U-Net do not provide suitable obstacles for planning, whereas shape regularized meshes lead to almost equal rates compared to ground truth planning. In successful cases, both U-Net and our approach achieve slightly lower but still acceptable mean minimal distances to risk structures due to the optimization step. The failure rate of 22% should be addressed in future work.

Table 1. Results on Dice and HD with mean(SD) for SegTHOR and MMWHS.
Full size table
Table 2. Quantitative results on planning metrics for SegThor and MMWHS.
Full size table

MMWHS: The evaluation on the MMWHS data set shows similar results. Table 1 again shows comparable Dice and Hausdorff scores clearly outperforming the U-Net approach. Having the same success rate of 90% (Table 2) but an even lower failure rate of 6% we conclude that shape regularization on deep learning solutions provides a promising approach for future endovascular procedures.

5 Discussion and Conclusion

We propose a complete preoperative planning pipeline for endovascular procedures, performing successive steps of segmentation, interactive problem definition and trajectory planning. Three key challenges - coherent boundaries extraction, interactive setup and optimal trajectory planning- are addressed by shape regularization for segmentation, minimal interaction and finally RRTs followed by convex optimization. Our experiments use twofold cross validation on two publicly available data sets. Here, we show in a functional evaluation that uses both segmentation (Dice, Hausdorff) and planning metrics (success & failure rate) that in data-scarce environments this shape regularization provides adequate preoperative planning for endovascular procedures.

Future endovascular interventions should automatize complex procedures whereas key parts of the surgery remain in full control of the clinician. To reach this goal, we aim to further reduce the violation rate by improving on both segmentation and trajectory quality. Hybrid loss guided networks [6] might boost the performance of our U-Nets. The initialization of ASMs could then better capture the inferior part of the descending aorta (Fig. 3) or the transitions of the right ventricle to pulmonary artery and right atrium. For planning, we believe that trajectory optimization in a RRT*-like fashion [11] might be more robust leading to higher clearance and thus also to lower failure rates.

This paper takes a critical first step toward an extensive preoperative pipeline which would eventually save the surgeon from high accumulated radiation dose.