Geodesic Tracking of Retinal Vascular Trees with Optical and TV-Flow Enhancement in SE(2)

Abstract

Retinal images are often used to examine the vascular system in a non-invasive way. Studying the behavior of the vasculature on the retina allows for noninvasive diagnosis of several diseases as these vessels and their behavior are representative of the behavior of vessels throughout the human body. For early diagnosis and analysis of diseases, it is important to compare and analyze the complex vasculature in retinal images automatically. In previous work, PDE-based geometric tracking and PDE-based enhancements in the homogeneous space of positions and orientations have been studied and turned out to be useful when dealing with complex structures (crossing of blood vessels in particular).

In this article, we propose a single new, more effective, Finsler function that integrates the strength of these two PDE-based approaches and additionally accounts for a number of optical effects (dehazing and illumination in particular). The results greatly improve both the previous left-invariant models and a recent data-driven model, when applied to real clinical and highly challenging images. Moreover, we show clear advantages of each module in our new single Finsler geometrical method.

1 Introduction

The retina allows for noninvasive examination of the vascular system since the vessels in the eye, and their corresponding behavior, are representative of the behavior of vessels throughout the rest of the body. Therefore, studying the behavior of the vasculature on the retina allows for noninvasive diagnosis of several diseases, like diabetes, hypertension, and Alzheimer’s disease [6, 16, 19]. Automatic vessel tracking algorithms help the efficient diagnosis of these diseases. Here, we rely on geodesic tracking methods which calculate the shortest path connecting two points on the same blood vessel, following the biological structure. We will show that the single geodesic model will also allow for acceptable tracking of full vascular trees on realistic retinal images of limited quality.

Diseases such as cataract disease give rise to cloudy retinal images [20], while camera movements lead to motion artifacts [14] and uneven illumination [24]. This affects the clarity and visibility of the vasculature in the images we want to track. To cope with the limitations in the quality of ophthalmology images in practice, we must integrate both contrast enhancement from optical image processing [22] and crossing-preserving contextual TV-flows, in our correct geodesic tracking of the vasculature, as we will show.

Fig. 1.

Geodesic tracking on the original image, (contrast-)enhanced image, and enhanced image after which TV-flow enhancement is done (left to right). The seeds and tips are indicated in resp. green and red. Yellow (/ red) circles indicate tracking mistakes that are (/ are not) fixed in the tracking of another column. (Color figure online)

Many approaches have been used when studying geodesic tracking methods on 2D images. However, in many methods, problems arise when dealing with difficult structures, like crossings and bifurcations, where standard geometric algorithms acting in the image domain $\mathbb {R}^2$ often take the wrong exit. Therefore one lifts the image to the homogeneous space of positions and orientations $\mathbb {M}_2$. In this lifted space, difficult structures are disentangled, cf. Fig. 2a. In previous works by various authors, it has been shown that PDE-based geometric tracking algorithms [4, 9, 10, 12] perform well in $\mathbb {M}_2$. Here, one first calculates a distance map which is based on the image data. After computing the distance map, the steepest descent algorithm is applied to find each shortest path from a tip (an endpoint) to the corresponding seed (a starting point).

In our tracking, we integrate PDE-enhancements, like crossing-preserving total variation flow (TV-flow) enhancement in $\mathbb {M}_2$ [18]. We will show this improves the results. Furthermore, optical enhancement [22] of limited-quality retinal images is required to keep equal contrast and intensity across the whole vasculature. This inevitably creates small noisy structures that are non-aligned with other structures in the data. Applying the TV-flow enhancement in $\mathbb {M}_2$ leads to crossing-preserving contextual denoising that preserves crossings, and line structures, and removes noisy non-aligned structures from the optical enhancement. Altogether the scheme results in a vascular tracking algorithm that provides better results as wavefronts follow the complex branching vasculature better than the approach in [4], see Fig. 1. Even a single geodesic front propagation (building the distance map initializing all seeds at the same time), where fronts follow the entire vasculature in one run produces good results, see Fig. 2.

Fig. 2.

Distance map built in $\mathbb {M}_2$ (2a; top) and isocontours of the minimum projection over the orientations back onto $\mathbb {R}^2$ (2a; bottom). The isocontours of the minimum projection of the distance map are constructed based on the original image and optically enhanced image after which TV-flow enhancement is done (in Fig. 2b and 2c resp.).

The main contributions of this article are; 1) the development of a new asymmetric, data-driven left-invariant Finsler geometric model that includes contextual contrast enhancement via TV-flows on SE(2), 2) experiments that show that application of this new Finsler geometric model reduces many tracking errors compared to previous left-invariant models [2, 9] and the recent data-driven model [4], 3) the new model performs very well on both realistic, unevenly illuminated retinal images and allows full vascular trees computations from a single distance map. The inclusion of the optical and TV-flow enhancements in the Finsler function no longer require a 2-step algorithm as in [4], but with the techniques in this work a single run of building the distance map suffices.

2 Lifted Space of Positions and Orientations $\mathbb {M}_2$

As explained in the introduction, it is beneficial to perform image processing (denoising, tracking) in the space of positions and orientations $\mathbb {M}_2$. This 3D space will offer us sufficient space to separate difficult (crossing) structures.

Definition 1

(Space of positions and orientations $\mathbb {M}_2$). The space of two-dimensional positions and orientations $\mathbb {M}_2$ is defined as a smooth manifold $\mathbb {M}_2:=\mathbb {R}^2\rtimes S^1$, where $S^1\equiv \mathbb {R}/(2\pi \mathbb {Z})\equiv SO(2)$ using the identification

$$\begin{aligned} \textbf{n}=(\cos \theta ,\sin \theta )\longleftrightarrow \theta \longleftrightarrow R_\theta \in SO(2), \end{aligned}$$

(1)

where $R_\theta$ is the counter-clockwise planar rotation over angle $\theta$. Elements in $\mathbb {M}_2$ are denoted by $(\textbf{x},\theta )\in \mathbb {R}^2\times \mathbb {R}/(2\pi \mathbb {Z})$. To stress the semidirect product of roto-translation group $SE(2):=\mathbb {R}^2\rtimes SO(2)$ acting on $\mathbb {M}_2$, we write $\mathbb {M}_2=\mathbb {R}^2\rtimes S^1$.

We lift the image from $\mathbb {R}^2$ to $\mathbb {M}_{2} \equiv SE(2)$ in order to separate crossing structures, using the orientation score transform.

Definition 2 (Orientation Score)

The orientation score transform $W_\psi f:\mathbb {L}_2(\mathbb {R}^2)\rightarrow \mathbb {L}_2(\mathbb {M}_2)$ using anisotropic wavelet $\psi$ maps an image $f\in \mathbb {L}_2(\mathbb {R}^2)$ to an orientation score $U=W_\psi f$ and is given by

$$\begin{aligned} (W_\psi f)(\textbf{x},\theta ):=\int _{\mathbb {R}^2}\overline{\psi (R_\theta ^{-1}(\textbf{y}-\textbf{x}))}\, f(\textbf{y})\; \textrm{d}\textbf{y}. \end{aligned}$$

In our experiments, we use cake wavelets [8, 18] as they do not tamper with data evidence and allow for fast reconstruction by integration over $\theta$.

In order to perform tracking in the lifted space of positions and orientations $\mathbb {M}_2$, we need to introduce a metric that is used to describe distances. This metric needs to satisfy the property that a roto-translation of the input yields the same roto-translation on the output.

Definition 3 (Left-Invariant Metric)

A metric tensor field $\mathcal {G}$ on $\mathbb {M}_2$ is called left invariant if

$$\begin{aligned} \mathcal {G}_{g\cdot \textbf{p}}\left( (L_g)_*\dot{\textbf{p}},(L_g)_*\dot{\textbf{p}}\right) =\mathcal {G}_\textbf{p}\left( \dot{\textbf{p}},\dot{\textbf{p}}\right) , \end{aligned}$$

for all $\textbf{p}\in \mathbb {M}_2$, all $\dot{\textbf{p}}\in T_\textbf{p}(\mathbb {M}_2)$, the tangent space to $\mathbb {M}_2$ at point $\textbf{p}$, and for all $g\in SE(2)$, where $L_g(\textbf{p})=g\cdot \textbf{p}=(\textbf{y},R_\alpha )\cdot (\textbf{x},\theta )=(\textbf{y}+R_\alpha \textbf{x},\alpha +\theta )$ with push-forward $(L_g)_*\dot{\textbf{p}} (U)=\dot{\textbf{p}}(U\circ L_g)$. More explicitly, $\dot{\textbf{p}}=\sum _{i=1}^3 \alpha ^i \left. \partial _{x^i}\right| _{\textbf{p}}$ with $\alpha ^i\in \mathbb {R},\; \textbf{p}=(x,y,\theta )$. Then

$$\begin{aligned} \left( (L_g)_*\dot{\textbf{p}}(U)\right)&=\alpha ^1(\cos \theta \partial _x+\sin \theta \partial _y)(U)(\textbf{p}) + \alpha ^2(-\sin \theta \partial _x+\cos \theta \partial _y)(U)(\textbf{p}) \\ {}&\qquad + \alpha ^3\partial _\theta (U)(\textbf{p}). \end{aligned}$$

More generally, a possibly asymmetric Finsler function defined on tangent-bundle $T(\mathbb {M}_2)=\{(\textbf{p},\dot{\textbf{p}}) \;|\; \dot{\textbf{p}} \in T_\textbf{p}(\mathbb {M}_2)\}$ given by $\mathcal {F}: T(\mathbb {M}_2) \rightarrow \mathbb {R}^+$ is left invariant if $\mathcal {F}(\textbf{p},\dot{\textbf{p}})=\mathcal {F}(g\cdot \textbf{p},(L_g)_*\dot{\textbf{p}})$ for all $(\textbf{p},\dot{\textbf{p}})\in T(\mathbb {M}_2), g\in SE(2)$.

Thereby the distance is left invariant:

for all $g \in SE(2)$, optimizing over the set $\varGamma _1$ of piecewise $C^1([0,1],\mathbb {M}_2)$-curves.

In our application, the asymmetric Finsler function will restrict backward movement as we will see in Sect. 3, and consequently cusps are avoided [9].

3 Existing Reeds-Shepp Car Models

Over the years, many geometric control problems have been proposed for geodesic tracking of blood vessels or vehicles. The ones closest to our model are the symmetric and asymmetric Reeds-Shepp car models. The symmetric Reeds-Shepp Car model, proposed in [3, 15], is given by

$$\begin{aligned} \mathcal {G}_\textbf{p}(\dot{\textbf{p}},\dot{\textbf{p}})=C(\textbf{p})^2\left( \xi ^2|\dot{\textbf{x}}\cdot \textbf{n}|^2+\frac{\xi ^2}{\zeta ^2}\Vert \dot{\textbf{x}}\wedge \textbf{n}\Vert ^2+\Vert \dot{\textbf{n}}\Vert ^2\right) , \end{aligned}$$

(2)

for all $\textbf{p}=(\textbf{x},\textbf{n})\in \mathbb {M}_2, \dot{\textbf{p}}=(\dot{\textbf{x}},\dot{\textbf{n}})\in T_\textbf{p}(\mathbb {M}_2)$ with $\Vert \dot{\textbf{x}}\wedge \textbf{n}\Vert ^2:=\Vert \dot{\textbf{x}}\Vert ^2-|\dot{\textbf{x}}\cdot \textbf{n}|^2$, where $\textbf{n}$ is constructed using the identification in (1). The asymmetric Reeds-Shepp Car model, proposed in [9], is given by the asymmetric Finsler norm/function

$$\begin{aligned} |\mathcal {F}({\textbf{p}},\dot{\textbf{p}})|^2= \mathcal {G}_\textbf{p}(\dot{\textbf{p}},\dot{\textbf{p}})+C(\textbf{p})^2(\varepsilon ^{-2}-1)\xi ^2|(\dot{\textbf{x}}\cdot \textbf{n})_-|^2, \end{aligned}$$

(3)

for all $\textbf{p}=(\textbf{x},\textbf{n})\in \mathbb {M}_2, \dot{\textbf{p}}=(\dot{\textbf{x}},\dot{\textbf{n}})\in T_\textbf{p}(\mathbb {M}_2)$ with $a_-:=\min \{0,a\}$. The parameter $\xi$ influences the flexibility of the tracking, weighing between spatial and angular movement. The anisotropy parameter $\zeta$ penalizes sideways movement. When $\zeta \downarrow 0$, the classical sub-Riemannian model appears.

The cost function $C:\mathbb {M}_2\rightarrow [\delta ,1]$, $\delta >0$, discourages movement outside the vascular structures via a crossing-preserving vesselness map $\mathcal {V}(\textbf{p})$ [11, 4, App.D]. Typically [3, eq.5.1] one has $C(\textbf{p})=(1+\lambda \mathcal {V}(\textbf{p})^p)^{-1}$. The choice of the cost function is important and in this article (Sect. 4, 5, 6) we propose to include illumination enhancement and crossing preserving TV-flow (prior to the vesselness map computation) in the cost function as this will greatly improve tracking results.

In the asymmetric Reeds-Shepp car model, the parameter $\varepsilon \in (0,1]$ describes how strongly the model has to adhere to the forward gear. When $\varepsilon =1$, we are in the symmetric model. When $\varepsilon \downarrow 0$, backward motion becomes prohibited.

Computation of shortest paths (geodesics) connecting seeds and tips is done in 2 steps. First, the distances to all points in the domain are calculated, resulting in a distance map. Then, the shortest path is obtained by a steepest descent algorithm applied on this distance map. In all experiments we use the Anisotropic Fast Marching [4, 13] by J.-M. Mirebeau for distance map computations.

Fig. 3.

Visualization of the physical model when imaging the retina; O is the actual image we would like to recover, $S_c$ is the perceived image (sum of purple and red reflected light). I stands for input illumination, $T_l$ and $T_{sc}$ resp. denote the transmission ratio of the lens and the intraocular scattering (incl. cataract). (Color figure online)

4 Illumination Enhancement

Previous approaches in retinal vessel tracking typically consider the unprocessed picture S taken by the ophthalmologist, e.g. [2, 9, 11]. However, this may deviate from the actual retinal image O which we aim to recover, due to possible cataract and uneven illumination. The physical model of the construction of the output image is visualized in Fig. 3. This yields the following standard optics formula:

$$\begin{aligned} S_c(\textbf{x})=L(\textbf{x})\left( T_{\text {sc}}^2(\textbf{x})O_c(\textbf{x})+1-T_{\text {sc}}(\textbf{x})\right) \text { with }L=I\cdot T_l^2, \end{aligned}$$

(4)

where c denotes the color channel in RGB or the luminance channel Y in YPbPr color space, cf. Wikipedia, and $L,T_{\text {sc}}:\varOmega \rightarrow [0,1]$ denote the illumination from outside the eye and transmission of the intraocular scattering respectively on domain $\textbf{x} \in \varOmega \subset \mathbb {R}^2$. The illumination from outside the eye L is composed of the input illumination I and the transmission ratio of the lens $T_l$. We apply an illumination correction, as done in [23]. After determining the illumination L, we re-express the Y channel (in YPbPr color space) of (4) to

$$\begin{aligned} \begin{array}{l} O_Y(\textbf{x})=\frac{L^{-1}(\textbf{x})S_Y(\textbf{x})-1+T_{\text {sc}}(\textbf{x})}{T_{\text {sc}}^2(\textbf{x})}\in [0,1], \text { for all }\textbf{x} \in \varOmega , \\ \text {with } T_{\text {sc}}=1-A\left( L^{-1}(\cdot )S(\cdot )+\sum \limits _{l=1}^n\frac{1}{n}\frac{1}{1+\exp (-\phi _l(\cdot ))}G_{\sigma _l} * \left( L^{-1}(\cdot )S(\cdot )\right) \right) , \end{array} \end{aligned}$$

(5)

with Gaussian kernel standard deviations $\sigma _l= \text {pixelsize}\cdot \, 2^{(l-1)}$ of the retinal image at scale level $l \in \{1,\ldots ,n\}$ where we took $n=4$ and where the sigmoids on scale coefficients above are included to control the range in [0, 1] and to allow for stable optimization below. The Y channel of the actual image O is obtained by solving the Euler-Lagrange equation of the Tikhonov regularization problem via the Karush-Kuhn-Tucker conditions including the constraints $O_Y\in [0,1]$:

$$\begin{aligned} (\phi ^{\min },A^{\min })= \text {argmin}_{(\phi ,A)}\, \{ \Vert O_Y-\overline{O_Y}\Vert ^2_{\mathbb {L}_{2}(\varOmega )}\!\!+\lambda \Vert \nabla O_Y\Vert ^2_{\mathbb {L}_{2}(\varOmega )}\} \end{aligned}$$

(6)

where we optimize w.r.t. $A>0$ and $\phi =(\phi _l)_{l=1}^n \in \mathbb {R}^n$, not $O_Y$. Here, $\overline{O_Y}\in \mathbb {R}$ is an estimation of the desired intensity level [23, Sec.3.5], and $\lambda$ regulates the smoothness of $O_Y$. After optimal non-constant $O_Y=O_Y(\phi ^{\min },A^{\min })$ is retrieved by (5), image O follows by linear conversion of YPbPr- to RGB-colors, via updated Pb- and Pr-channels.

5 TV-Flow Enhancement

TV-flow enhancement is a valuable technique to denoise surfaces, but at the same time preserve sharp edges. Recall that the metric intrinsic gradient is given by

$$\begin{aligned} \mathbb {M}_2\ni \textbf{p}\mapsto \nabla _{\mathcal {G}} \phi (\textbf{p})=\left( \left( \mathcal {G}\right) ^{-1}\textrm{d}\phi \right) \!\left( \textbf{p}\right) \in T_\textbf{p}(\mathbb {M}_2), \end{aligned}$$

using $\mathcal {G}$ in (2) with $\zeta =C\xi =1, C=10$. Then TV-flow $U\mapsto W_0(\cdot ,t)$ is given by

$$\begin{aligned} {\left\{ \begin{array}{ll} \frac{\partial W_\varepsilon }{\partial t}(\textbf{p},t)=\text {div}\!\left( \frac{\nabla _{\mathcal {G}} W_\varepsilon (\cdot ,t)}{\varepsilon ^2+(\nabla _{\mathcal {G}} W(\cdot ,t))^2}\right) \!(\textbf{p}), \qquad \textbf{p}\in \mathbb {M}_2, \ t \ge 0, \\ W_\varepsilon (\textbf{p},0)=U(\textbf{p}) \end{array}\right. } \end{aligned}$$

and $W_0(\textbf{p},t)=\lim _{\varepsilon \downarrow 0}W_\varepsilon (\textbf{p},t) .$ For proof of the $\mathbb {L}_2$-convergence, see [18]. Training of the end-time t of the TV-flow is not needed as for all lifted optically enhanced (cf. Sect. 4) images U of the STAR-dataset [21], end-time $t=0.5$ is nearly optimal for subsequent tracking, and $\varDelta t=0.1$ always remains in the stability region [18]. The same settings provided optimal PSNR-ratios in [18].

6 A Finsler Metric on $\mathbb {M}_2$ that Includes the Enhancements

Our goal was to track vasculature accurately. In order to do so, one needs a metric tensor field that describes distances on the manifold. In some cases, it is beneficial to construct a metric tensor field $\mathcal {G}^U$ that depends explicitly on the underlying orientation score data U. This “data-driven” metric tensor field needs to be left invariant with respect to the roto-translation of the underlying data:

Definition 4 (Data-Driven Left-Invariant Metric (DDLIM))

The metric tensor fields $\mathcal {G}^U$ and $\mathcal {F}^{U}$ on $\mathbb {M}_2$ are data-driven left invariant when they satisfy for all $(\textbf{p},\dot{\textbf{p}})\in T(\mathbb {M}_2)$ and all $g\in SE(2)$:

$$\begin{aligned} \mathcal {G}_\textbf{p}^U(\dot{\textbf{p}},\dot{\textbf{p}})=\mathcal {G}_{g\cdot \textbf{p}}^{\mathcal {L}_gU}((L_g)_*\dot{\textbf{p}},(L_g)_*\dot{\textbf{p}}), \text { and } \mathcal {F}^{U}(\textbf{p},\dot{\textbf{p}})= \mathcal {F}^{\mathcal {L}_gU}( g\cdot \textbf{p}, (L_{g})_* \dot{\textbf{p}}), \end{aligned}$$

(7)

where $\mathcal {L}_gU(\textbf{h}):=U(L_{g^{-1}}\textbf{h}):=U(g^{-1}\cdot \textbf{h})$ for all $\textbf{h}\in \mathbb {M}_2$.

The considered data-driven left-invariant metric tensor fields are given by

$$\begin{aligned} \mathcal {G}_\textbf{p}^U(\dot{\textbf{p}},\dot{\textbf{p}})&=\mathcal {G}_\textbf{p}(\dot{\textbf{p}},\dot{\textbf{p}})+\mu \, C^2(\textbf{p}) \frac{\Vert HU|_\textbf{p}(\dot{\textbf{p}},\cdot )\Vert _*^2}{\max \limits _{\Vert \dot{\textbf{q}}\Vert =1}\Vert HU|_\textbf{p}(\dot{\textbf{q}},\cdot )\Vert _*^2},\end{aligned}$$

(8)

$$\begin{aligned} \left| \mathcal {F}^U(\textbf{p},\dot{\textbf{p}})\right| ^2&=\left| \mathcal {F}(\textbf{p},\dot{\textbf{p}})\right| ^2+\mu \, C^2(\textbf{p})\frac{\Vert HU|_\textbf{p}(\dot{\textbf{p}},\cdot )\Vert _*^2}{\max \limits _{\Vert \dot{\textbf{q}}\Vert =1}\Vert HU|_\textbf{p}(\dot{\textbf{q}},\cdot )\Vert _*^2}, \end{aligned}$$

(9)

for all $\textbf{p}=(\textbf{x},\textbf{n}), \dot{\textbf{p}}=(\dot{\textbf{x}},\dot{\textbf{n}})$, representing the symmetric and asymmetric metric tensor fields respectively. In (8) and (9), the metric tensor fields $\mathcal {G}$ and $\mathcal {F}$ were introduced in (2) and (3) respectively. The Hessian field HU is defined as $HU:=\nabla (\textrm{d}U)$, w.r.t. plus Cartan connection $\nabla ^{[+]}$ for computational details see [7, 4, Rem.8], and $\Vert \cdot \Vert _*$ denotes the dual norm w.r.t. $\sqrt{\mathcal {G}_\textbf{p}(\dot{\textbf{p}},\dot{\textbf{p}})}$, where $\zeta =\xi =C=1$.

The parameter $\mu >0$ regulates the inclusion of the new data-driven term, and $C(\textbf{p})$ denotes the cost function described in [4, App. D].

Remark 1

The construction of this cost now relies on the orientation score U of the optically enhanced image with TV-flow enhancement, whereas previously it relied on the orientation score of the unprocessed image. Akin to [4, App.C], one can show that the new Finsler/Riemannian metric tensor fields (8) are DDLIM.

The geodesics are calculated by steepest descent on distance maps using a metric that describes distances in $\mathbb {M}_2$.

Definition 5 (Data-Driven Riemannian Distance)

The data-driven Riemannian distance $d_{\mathcal {G}^U}$ from a point $\textbf{p}\in \mathbb {M}_2$ to a point $\textbf{q}\in \mathbb {M}_2$ is given by

$$\begin{aligned} d_{\mathcal {G}^U}(\textbf{p},\textbf{q})= \underset{{\begin{array}{c}\gamma \in \varGamma _1,\\ \gamma (0)=\textbf{p}, \\ \gamma (1)=\textbf{q}\end{array}}}{\inf } \int _0^1\sqrt{\mathcal {G}_{\gamma (t)}^U\left( \dot{\gamma }(t),\dot{\gamma }(t)\right) }\;\textrm{d}t \end{aligned}$$

(10)

where $\varGamma _1:=\{\gamma :[0,1]\rightarrow \mathbb {M}_2|\gamma \in PC^1([0,1],\mathbb {M}_2)\}$ with $PC^1$ the space of piecewise continuously differentiable curves in $\mathbb {M}_2$, and $\dot{\gamma }(t):=\frac{\textrm{d}}{\textrm{d}t}\gamma (t)$. The quasi-distance that belongs to the asymmetric Finslerian model (9) is given by

(11)

Lemma 1

If $\mathcal {F}^U$ is DDLIM (Definition 4) then distance $d_{\mathcal {F}^U}$ satisfies:

$$\begin{aligned} \forall _{g \in SE(2)} \forall _{\textbf{p}_{1},\textbf{p}_2 \in \mathbb {M}_2}\ : \ d_{\mathcal {F}^{\mathcal {L}_{g}U}}(g \cdot \textbf{p}_1, g \cdot \textbf{p}_2)= d_{\mathcal {F}^U}(\textbf{p}_1, \textbf{p}_2). \end{aligned}$$

(12)

Proof

One has

$=d_{\mathcal {F}^U}(\textbf{p}_1,\textbf{p}_2)$, where $g^{-1}\cdot \gamma \in \varGamma _1 \Leftrightarrow \gamma \in \varGamma _1$, from which (12) follows. $\Box$

The shortest curves are computed using steepest descent on the distance map, departing from tip $\textbf{p}\in \mathbb {M}_2$ towards seed $\textbf{p}_0 \in \mathbb {M}_2$ as described in Theorem 1.

Theorem 1

The shortest curve $\gamma :[0,1] \rightarrow \mathbb {M}_2$ with $\gamma (0)=\textbf{p}$ and $\gamma (1)=\textbf{p}_0$ can be computed by steepest descent tracking on distance map $W(\textbf{p})=d_{\mathcal {F}^U}(\textbf{p},\textbf{p}_0)$

$$\begin{aligned} \gamma (t):=\gamma ^U_{\textbf{p},\textbf{p}_0}(t)= \text {Exp}_{\textbf{p}}(t\, v(W)), \qquad t \in [0,1], \end{aligned}$$

(13)

where Exp integrates the following vector field on $\mathbb {M}_2$: $v(W):=-W(\textbf{p})\nabla _{\mathcal {F}^{U}}W$ and where W is the viscosity solution of the eikonal PDE system

$$\begin{aligned} {\left\{ \begin{array}{ll} \mathcal {F}_{U}^{*}(\textbf{p}, \textrm{d}W(\textbf{p}))=1 \qquad \textbf{p}\in \mathbb {M}_2,\\ W(\textbf{p}_0)=0, \end{array}\right. } \end{aligned}$$

(14)

assuming $\textbf{p}$ is neither a 1st Maxwell-point nor a conjugate point, with dual Finsler function $\mathcal {F}_{U}^{*}(\textbf{p},\hat{\textbf{p}}):= \max \{ \langle \hat{\textbf{p}},\dot{\textbf{p}} \rangle |\; \dot{\textbf{p}} \in T_{\textbf{p}}(\mathbb {M}_2) \text { with } \mathcal {F}^U(\textbf{p},\dot{\textbf{p}})\le 1 \}$. As v(W) is data-driven left invariant, the geodesics carry the symmetry

$$\begin{aligned} \gamma _{g\cdot \textbf{p}, g\cdot \textbf{p}_0}^{\mathcal {L}_gU}(t) = g\;\gamma _{\textbf{p},\textbf{p}_0}^U(t) \text { for all }g\in SE(2),\textbf{p},\textbf{p}_0 \in \mathbb {M}_2, t \in [0,1]. \end{aligned}$$

(15)

Proof

This is a special case of [4, Thm.1] with Lie group $SE(2) \equiv \mathbb {M}_2$. Then this yields the symmetric case $\Vert \nabla _{\mathcal {G}^U}W(\textbf{p})\Vert =1$ in the usual eikonal PDE form. Inclusion of the asymmetric front propagation (relying on asymmetric Finsler metric $\mathcal {F}^U$) requires a replacement of $\Vert \nabla _{\mathcal {G}^U}W(\textbf{p})\Vert =1$ with a dual norm expression $\mathcal {F}_{U}^{*}(\textbf{p}, \textrm{d}W(\textbf{p}))=1$, where one takes the positive part of the spatial momentum component in direction $\cos \theta \textrm{d} x +\sin \theta \textrm{d}y \in T^*(\mathbb {M}_2)$. This is similar to the technique in [9, Thm.4] but due to the data-driven behavior $\mathcal {F}^U$ this is subtle [4, Eq. 43, Lem. 3] and also directly applies to our model (16) using cost function C (incl. optical & TF-flow enhancement) as explained in Remark 1. Also, the backtracking requires a subtle adaptation: instead of ordinary intrinsic gradient descent in direction $\nabla _{\mathcal {G}^U}W=(\mathcal {G}^{U})^{-1}\textrm{d}W$ it now becomes more general descent in direction $\nabla _{\mathcal {F}^U}W(\cdot ):=\textrm{d}\mathcal {F}_{U}^*(\cdot , \textrm{d}W(\cdot ))$ as explained in [9, prop.4].    $\square$

In the experimental section, we rely on the mixed metric tensor field, which is needed to avoid wrong exits at complex structures, see [4], and is given by:

$$\begin{aligned}&\mathcal {G}_\textbf{p}^{M}(\dot{\textbf{p}},\dot{\textbf{p}})=\kappa (\textbf{x})\,\mathcal {G}_\textbf{p}(\dot{\textbf{p}},\dot{\textbf{p}})+(1-\kappa (\textbf{x}))\; \mathcal {G}_\textbf{p}^{U}(\dot{\textbf{p}},\dot{\textbf{p}}), \ \textbf{p}=(\textbf{x},\textbf{n}) \in \mathbb {M}_2 \nonumber \\&\mathcal {F}^{M}(\textbf{p},\dot{\textbf{p}})^2=\kappa (\textbf{x})\, \mathcal {F}(\textbf{p},\dot{\textbf{p}})^2+(1-\kappa (\textbf{x}))\, \mathcal {F}^{U}(\textbf{p},\dot{\textbf{p}})^2, \end{aligned}$$

(16)

with $\kappa = \mathbbm {1}_A * G_{\sigma }$ and A the crossing structure locations and Gaussian $G_{\sigma =1\textrm{pix}}$.

7 Experimental Results

We rely on the asymmetric metric tensor field (16) to calculate the geodesics of the 3 different models. These models are constructed based on a) the picture taken by the ophthalmologist (original image), b) the original image with illumination enhancement as explained in Sect. 4 (optically enhanced image), and c) the optically enhanced image with crossing-preserving TV-flow enhancement discussed in Sect. 5. In this section, we illustrate the results for a specific image and refer to Table 1 for an overview of the performance of the different models on the STAR dataset [1, 21]. These results are consistent with the discussed example, and for reproducible code and all processed images see [5]. In all experiments we set standard parameter settings [3, eq.5.1], [4, eq.65] for cost-function C ($p=3,\lambda =1000$), for the metrics ($\xi =\zeta =0.1$) in (2), for TV-flow ($t=0.5$).

In prior research, the “original image” directly entered the metric tensor field on $\mathbb {M}_2$ when calculating the geodesics. These images are of varying quality, depending on the patient’s condition and the used equipment. Applying a tracking algorithm, like Anisotropic Fast Marching [4], on the metric tensor field based on the original image, often results in tracking mistakes due to uneven illumination, both along vascular structures and on the background, making it hard to distinguish different structures. In Fig. 1a, one sees the tracking results on an original, unevenly illuminated, non-enhanced image, where all vessels were tracked in one single run. At a lot of locations (13), the tracking connects the seeds $\textbf{p}_0$ and tips $\textbf{p}$ incorrectly. The optical enhancement explained in Sect. 4 corrects for uneven illumination. Calculating the geodesics using the metric tensor field relying upon the optically enhanced image, reduces the number of tracking mistakes significantly (to 5), cf. Fig. 1b. Due to pointwise optimization in the optical enhancement, noise is generated. The crossing-preserving total variation flow enhancement suppresses this noise and indeed results in even fewer tracking mistakes (3), cf. Fig. 1c.

Fig. 4.

Tracking of Vascular Tree per Vessel Type: Tracking with the mixed model $(\mathbb {M}_2,\mathcal {F}^M)$ proposed in (16) with $\mu =15$. Prior classification of vascular trees by type (artery/vein resp. white/cyan) only results in perfect tracking of the vessel tree on the enhanced images. Yellow circles indicate tracking mistakes.

Calculating the tracking results in Fig. 1 uses no knowledge about the vasculature, apart from seed and tip locations. One might incorporate prior knowledge A) on vessel types (artery/vein), or B) on the connectivity of tips and seeds.

We start by investigating prior knowledge on A), where we first connect all tips on arteries to the seeds on arteries, and similarly for the tips and seeds on veins. Figure 4 shows that the tracking results improve significantly for all cases; the number of tracking mistakes at crossings reduces from (13, 5, 3) to (5, 0, 0) for resp. the original, optically enhanced image excl. and incl. TV-flow enhancement.

Second, we investigate the prior knowledge on B). In Fig. 5, the tracking results connecting the tips to their corresponding seed are presented. The number of correct tracks has improved once again, to only (3, 0, 0) mistakes for resp. the original, optically enhanced and optically enhanced with TV-flow image.

We report the tracking results for the three different approaches on images from STAR [1, 21], in particular the example in Fig. 1, 4, and 5. We observe the same trend in performance for other images which we summarize in Table 1. We calculate the (weighted) percentage of incorrectly calculated geodesics by means of:

$$\begin{aligned} \epsilon :=\frac{1}{|T|}\sum _{\textbf{y}\in T}\left( 1-\frac{\Vert \textbf{y}-\textbf{x}_0(\textbf{y})\Vert }{\sqrt{N_x^2+N_y^2}}\right) C_0(\textbf{x}(\textbf{y}),\textbf{y})\ge 0, \end{aligned}$$

(17)

Here S, T denote the sets of resp. seeds (near the optic disk) and tips. The image size is $N_x\times N_y$. The ground truth seed and calculated seed (first arriving front in the distance map, cf. Fig. 2) corresponding to the tip $\textbf{y}$ are resp. denoted by $\textbf{x}_0(\textbf{y}) \in S$ and $\textbf{x}(\textbf{y}) \in S$. Function $C_0: S \times T \rightarrow \{0,1\}$ is given by $C_0(\textbf{x},\textbf{y})=0$ if the tracking between $\textbf{x}$ and $\textbf{y}$ is correct and $C_0(\textbf{x},\textbf{y})=1$ otherwise.

Fig. 5.

Tracking of Vascular Tree per Seed on the Optic Disk: Tracking with the mixed model $(\mathbb {M}_2,\mathcal {F}^M)$ proposed in (16) with $\mu =15$. Prior grouping of tips (in red) and seeds (in green) only results in perfect tracking of the vessel tree on the enhanced images. Tracking mistakes are indicated by yellow circles. (Color figure online)

Table 1. Error measure $\epsilon$ of each tracking applied to STAR images in [5], calculated by (17). Highlighted: best results per tracking.

We evaluate with a harsh error measure (17): one crossing mistake (indicated by a circle) often causes more errors, when the vessel bifurcates after the crossing.

The optical enhancement and TV-flow regularization applied on the original images, result in more accurate geodesics compared to those calculated directly on the original images, as can be seen in Table 1. The more prior information we use, the more accurate the geodesics follow the vasculature. Remarkably, tracking requiring artery-vein classification of seeds and tips performs similarly and is easier to automate than tracking with knowledge of seed-tip connectivity.

Conclusion

We developed a new asymmetric, data-driven left-invariant Finsler geometric model that includes contextual contrast enhancement via TV-flows on SE(2). Experiments reveal that application of this new Finsler geometric model has benefits over previous left-invariant models [2, 9] and the recent data-driven model [4]. The new model reduces many errors and performs very well on both realistic and challenging low-quality retinal images where full vascular trees are computed from a single asymmetric Finslerian distance map. Although we have shown that both the contrast enhancement and the TV-flow on SE(2) in the new Finslerian model are highly beneficial, there are still exceptional cases where vessel tracts take the wrong exit. This happens at places where both a crossing and a bifurcation occur (cf. the red circles in Fig. 1c). Therefore, in future work, we aim to tackle these cases by automatic artery vein classification via PDE-G-CNNs [17], as our experiments show this allows us to obtain the same good practical results as with the ‘tracking per seed’ (that requires too costly user-knowledge).