Keywords

1 Introduction

The diagnosis of coronary heart diseases needs a 3D model of the coronary artery with precisely labeled vessels, which provides necessary information for many subsequent procedures, such as locating stenosis/occlusion, generating medical reports, calculating statistics for important anatomical branches and visualizing regions of interest [1, 2]. As shown in Fig. 1, coronary arteries are intertwined with or attached to multiple organs, i.e., the heart chambers, aorta, and heart surface. Precisely labeling them is practically challenging, mainly due to 1) coronary vessels are typically tortuous dendritic tubular structures with diverse lengths; 2) coronary vessels have dramatically changed shape appearance across individuals and spatial positions.

Fig. 1.
figure 1

Sample heart consisting of aorta, atrium, ventricles and coronary arteries. (a) A chamber along with major coronary arteries; (b) A 3D heart shows the blood vessels intertwining and attaching around the organ; (c) The coronary arteries structure varies among individuals, as well as the thin and messy side branches.

Full size image

Several automatic methods have been proposed for coronary artery (or similar 3D structure) labeling, e.g., based on registration techniques, CNNs, and point-cloud deep learning. Traditional methods [3, 4] adopt registration techniques to label coronary arteries via main branch matching, which cannot sensitively identify detailed vessels. CNN-based methods can perform labeling more efficiently, while general convolutional kernels defined in the grid space are hard to capture the contextual information along with the tortuous tree-like structures of the coronary vessels [5]. Inspired by the pioneering works in 3D computer vision and graphics [6,7,8], more recent works attempted to apply point-cloud networks or graph convolutional networks (GCNs) for tree-like vascular structures labeling, achieving promising resulting in specific applications [9,10,11,12]. However, since coronary vessels have relatively more complicated structures, deep networks working solely with point-coordinates information may fail to capture discriminative shape details for fine-grained vessel labeling.

In this paper, to identify detailed coronary vessels with varying lengths and shape appearance, we propose to leverage comprehensive spatial/anatomical dependencies as explicit guidance in developing task-specific point-cloud deep networks. That is, a CorLab-Net method is designed to integrate both inter-organ and intra-artery contextual information with point-coordinate information for accurate point-wise labeling. Our CorLab-Net has three key technical contributions: 1) It defines a spatial distance field between vessels and heart chambers, serving as the inter-organ spatial guidance to assist the localization of the blood vessels. 2) It predicts a morphological distance field between vessel points and key joint points (i.e., landmarks), providing the intra-artery anatomical guidance to help identify different vessel branches at the junctions. 3) It integrates GCN modules to capture the high-order correlations between neighboring points, rendering the intra-artery local guidance to enhance labeling consistency and correct prediction outliers, especially at the vascular endings.

2 Method

Our CorLab-Net method mainly includes three components: anatomical distance field module, morphological distance field module, and a point-cloud network integrating GCN modules, such as the schematic diagram shown in Fig. 2. The two distance field modules form an effective enriched-feature space by capturing the inter-organ and intra-artery spatial dependencies to provide detailed contextual information. Based on the outputs of these two modules, the point-cloud network uses PointNet++ [7] as the backbone and integrates the GCN modules in its decoding part to conduct locally-consistent point-wise labeling.

Fig. 2.
figure 2

Overview of the proposed framework. (a) Entire processing pipeline of our network; (b) The point cloud network learns the principle of consistency between neighboring points based on PointNet++ and GCN module.

Full size image

2.1 Anatomical Distance Field

As can be seen from the examples in Fig. 1, coronary vessels start from the aorta, wrap around and attach to the surfaces of the left and right atria and ventricle. In practice, clinicians annotate different artery vessels according to the spatial associations between the artery and specific surrounding organs. It implies that the inter-organ anatomical dependencies convey critical contextual information to help locate different vessels. Unfortunately, such anatomical dependencies were typically ignored by existing automatic methods [10, 13], as they work solely on point coordinates for blood vessel identification.

Inspired by the clinical experience, we assume each branch of a coronary artery is roughly distributed at a fixed position of the heart, according to which an anatomical distance field is defined to model the inter-organ dependencies between different coronary branches and surrounding heart parts. Specifically, we denote the points of coronary arteries P as \(P=[p_1,p_2,\cdots p_N ]\) with \(N=2048\) points, \(p_i\in R^3\), and surface points of heart chambers C as \(C=[c_1,c_2,\cdots c_M ]\) with \(M\in [8000,14000]\) points, \(c_j\in R^3\). The details to calculate such a distance field are as follows:

$$\begin{aligned} S_{i}=\min _{i=1, \cdots , N \atop j=1, \cdots , M} \sqrt{\left( p_{i, 1}-c_{j, 1}\right) ^{2}+\left( p_{i, 2}-c_{j, 2}\right) ^{2}+\left( p_{i, 3}-c_{j, 3}\right) ^{2}}~, \end{aligned}$$
(1)
$$\begin{aligned} Dist_i = {\left\{ \begin{array}{ll} \frac{S_i}{Th},~&{}if~\frac{S_i}{Th}<1 ~,\\ 1, ~&{}else ~,\\ \end{array}\right. } \end{aligned}$$
(2)

where Th is an empirical number, and \(Th=50\) in this study.

In practice, we calculate the minimum distance from each point of the coronary artery branch to the surfaces of the aorta, left and right atria and left and right ventricle, respectively, and then merge these distance constraints with point coordinate information to form the whole distance field, as shown in Fig. 2. Finally, these discriminative information and other effective vectors are fed into point cloud network to enrich the whole input feature space.

2.2 Morphological Distance Field

Traditional point-cloud networks treat all input points equally [6, 7], ignoring the important contributions of key points in defining the shape of an input point cloud. For example, in coronary labeling, the identification of a joint point can provide strong clues regarding the semantic labels of all other points, from the proximal one to the distal one, on the corresponding artery branch.

To integrate such intra-artery contextual information into our CorLab-Net method, a morphological distance field is defined accordingly. Specifically, we assume that different points have different contributions to the network; since joint points contain more semantic information, the network should pay more attention to them. Therefore, as shown in Fig. 2, we first apply a landmark-detection point-cloud network to the original coronary data to regress the joint points of each branch. Then, the spatial displacements from each point on the branch to the respective joint points are further calculated, forming the morphological distance field.

In the implementation process, we directly regress the distance vectors from each point to the closest joint points to replace the above process. Note that these two processes are equivalent. Specifically, the input to this module is coronary artery points P. The output Q are the vector differences between each point and the closest joint point of sub-branches, where \(Q=[q_1,q_2,\cdots q_K ]\) with \(K=2048\) points, \(q_r\in R^3\). To train the landmark regression network, we define our regression loss based on the chamfer distance [14, 15] to constrain the predicted vectors Q to be close to the real vector differences T. The regression loss \(L_{reg}\) for network training is the chamfer distance between the predicted vector differences Q and ground-truth vectors T:

$$\begin{aligned} L_{reg}(Q,T) = \frac{1}{|Q|}\sum \limits _{q_k \in Q}\min \limits _{t_w \in T}{\Vert q_k-t_w\Vert ^2_2} + \frac{1}{|T|}\sum \limits _{t_w \in T}\min \limits _{q_k \in Q}{\Vert t_w-q_k\Vert ^2_2}~, \end{aligned}$$
(3)

2.3 Point-Cloud Deep Network

Since there is no order information in the point cloud data, the network captures context information between neighboring points by using various aggregating neighboring points in the encoder stage, such as Multi-Scale Grouping (MSG) and Multi-Resolution Grouping (MRG) [7]. The decoder stage is often overlooked. It generates high-level semantic vectors, which has a strong mapping relationship with the label.

In labeling task of coronary arteries, the neighboring points generally belong to the same category, especially the points at vascular endings. If the network can match the label relationship of neighboring points with its corresponding high-level features, the labeling consistency relationship will be established through spatial distribution between each point and neighboring points. The consistency can better guide the network to make decisions.

We build a graph structure to capture the consistent relationship between neighboring points, vertices of graph are 3D coordinates of point cloud, and edges are determined by Euclidean distance between the coordinates. If two points are close together in space, they are connected, and the threshold d is set to measure whether the two vertices are connected.

In this way, the high-level features of all points in the neighborhood are fused by the graph structure. The features of each points have a strong mapping relationship with the label, and the fused features have enriched feature representation.

3 Experiments and Results

3.1 Dataset and Evaluation Metrics

In our experiments, we collected a private database of 100 subjects, and all of them are annotated by three experts. Our data included not only voxel label for each coronary artery vessel, but also labels for atrium and ventricles. These subjects and corresponding annotations compose the experimental dataset.

The evaluation is performed on coronary artery segments by the predicted label and ground truth. The precision rate for each branch is calculated by P \(=\frac{\text {TP}}{\text {TP+FN}}\). The Recall is Recall \(=\frac{\text {TP}}{\text {TP+FP}}\). The F1 score is F1 \(=2*\frac{ \text{ Precision } \times \text{ Recall } }{ \text{ Precision } + \text{ Recall } }\). Since the numbers of coronary branches in different subjects are imbalance, we also adopt the mean metrics of all branch segments.

3.2 Implementation Details

Point Cloud Processing. Due to the differences of subjects, our image sequence is not fixed, and the size is \(512\times 512\times S_e\), where \(S_e \in [155,353]\). Therefore, before the image is sampled into point cloud, it needs to be normalized. There are mainly three steps in normalization: (1) the input image is cropped according to the position of the segmented coronary artery in the image, and only the part containing coronary vessels is retained. (2) Since the cropped sizes of each subject are inconsistent, the cropped image coordinates are divided by the size for normalization. (3) The 3D image is sampled as point cloud, and the point cloud data has been normalized to the range [0, 1].

Parameters Setting. The numbers of vertices H in the two GCNs are set to 512 and 1024, respectively, and GCN parameters are the same as [16]. The threshold d for edge connection is an empirical number, the two GCNs are set to 0.1 and 0.05 in this study, respectively. Five-fold cross-validation is used in the experiment, and we train the network for 200 epochs using Adam optimizer with an initial learning rate of 0.001. The model is implemented using PyTorch with an NVIDIA Tesla V100S GPU, and each mini-batch contains 4 point cloud samples. In addition, chamfer distance and cross-entropy are, respectively, used as loss functions in regression and classification network.

3.3 Comparison with Existing Methods

To verify the effectiveness of our framework, several point cloud algorithms were tested on dataset, including PointNet [6], PointNet++ [7] and HN-Net [10]. They all directly take the 3D point cloud as input to predict the labels of coronary arteries. The statistic results and visual presentations are presented in Table. 1 and Fig. 3, respectively.

Table 1. Comparisons of PointNet (PN), PointNet++ (PN++), HN-Net (HN) and CorLab-Net on our dataset. Recall, Precision(P) and F1 score are used as the evaluation metrics (%).
Full size table

Quantitative Results. The overall coronary labeling results are summarised in Table.1, where the proposed framework shows significant improvement to other methods. Concretely, compared with the backbone PointNet++, our method increases by 11%, 10.6% and 11.3% in meanPrecision, meanRecall and F1 score, respectively, which demonstrates the effectiveness of the anatomical dependency-aware point-cloud learning framework. At the same time, it can be seen that the metrics of PointNet and HN-Net are far below our method, which further indicates the advantages of all proposed module in our framework.

Qualitative Results. The visual presentations are shown in Fig. 3 for three typical subjects. It can be observed that labeling results produced by our method match better with the ground truth, especially for typical parts, such as vascular bifurcation and vascular endings (as highlighted by red arrows in Fig. 3).

Fig. 3.
figure 3

Comparison of labeling results for three typical subjects by four different methods. Different colors represent different categories of coronary vessels, and there are 11 classes in total. The red circle with arrow are artificially designed for visual presentation, highlighting the typical parts. (Color figure online)

Full size image

The first column is the ground truth, followed by the comparison result of our method and others. Notably, the other three methods not only fail to produce satisfactory results in these difficult areas, but also have a large number of label errors in the main branches. Our method can avoid this problem because it adds enriched feature vectors and point consistency constraint. The qualitative results shown in Fig. 3 are consistent with the quantitative comparison, which further demonstrates the effectiveness and efficiency of our framework for automatic labeling of coronary arteries.

3.4 Ablation Study

We conduct extensive experiments to validate the effectiveness of our model components. First, we build a baseline network, denoted as bNet, which is our network backbone (PointNet++) for automatic labeling of coronary arteries. All the alternative networks are derived by augmenting the baseline network with different network components. We describe the details and results in the following.

Anatomical Distance Field. The anatomical distance field represent relative location of vessels and chambers, which constrains the vessel execution at the region level. To validate its benefits, we augment the baseline network bNet with the anatomical distance field (bNet-A) and compare prediction results of both networks in Table 2. By appending the module, the meanPrecision, meaRrecall and meanF1 increase by 5.5%, 4.2%, 5.5%, respectively.

Morphological Distance Field. The morphological distance field focuses on the important contribution of key points to the network, which represents the contextual relationships of points at the part level. To validate its benefit, we add the module to labeling vessel (bNet-A-M), and the metrics go up over 3.6%.

Point Consistency based on GCN. To further minimize the errors of network at vascular endings, we add the graph network module to capture the category consistency of the neighboring points, which belongs to the constraint relationship of the point level. We have verified all modules on the dataset and the metrics have been improved with varying degrees.

Table 2. Part of the ablation study results for our method. The anatomical distance field, morphological distance field, as well as point consistency based on GCN are all essential parts of our CorLab-Net.
Full size table

4 Conclusion

In this study, we propose an anatomical dependency-aware point-cloud learning framework (CorLab-Net) for coronary artery labeling. Specifically, we develop three modules, including anatomical distance field, morphological distance field, and point consistency block, to accurately label the coronary arteries. Compared with the existing point cloud networks, our method achieves superior performance and outperforms the state-of-the-art performance by a large margin, which suggests the potential applicability of our framework in real-world clinical scenarios. Furthermore, our work addressed an essential task in clinical practice. Automatic and accurate labeling of the coronary artery has a great benefit on the diagnosis of coronary heart diseases, which provides necessary information for many subsequent procedures, such as generating medical reports, calculating statistics for important anatomical branches, and visualizing regions of interest.