Landmark-Constrained Statistical Shape Analysis of Elastic Curves and Surfaces

Abstract

We present a framework for landmark-constrained elastic shape analysis of curves and surfaces. While most of statistical shape analysis focuses on either landmark-based or curve-based representations, we describe a new approach that is able to unify them. The described method has its roots in elastic shape analysis, where elastic metrics are used for comparisons and statistical analysis of shapes. The landmark information is incorporated into the shape representation via hard constraints in the re-parameterization group. We use the square-root velocity function representation for curves and square-root normal field representation for surfaces to greatly simplify the implementation of these methods. We give multiple complex examples from graphics and computer vision, wherein the landmark-constrained shape analysis framework is able to provide natural deformations between shapes as well as representative summaries, including the average and principal directions of variability, of samples of shapes.

1 Introduction

Shape is a fundamental property of objects observed in images, and is often defined as the appearance of their outlines. In the case of two dimensional images, the outlines of objects form planar open and closed curves. In the case of three dimensional images, such as medical ones including magnetic resonance images (MRIs), the outlines of structures form surfaces. Due to improvements in imaging technology, shape datasets have become ubiquitous in many different applications including biology, medicine, biometrics, graphics, bioinformatics, and many others. As a result, statistical shape analysis is an emerging discipline within statistics that seeks to make inferences about a population of objects, represented by their corresponding shapes. To develop statistical procedures applicable to shapes, one must first represent them mathematically; this is not a simple task. Consider a lightbulb, for example. The shape of a lightbulb is easily recognizable. However, it is important to note that if the lightbulb is moved to a different location in the image, rotated, or re-scaled, its shape does not change. Thus, mathematically, shape is a property of an object, which is invariant when the object is translated in space, rotated, or re-scaled. Because of these required invariances, new tools for analyzing shapes are required. That is, standard univariate or multivariate statistical methods are often not directly applicable in this context, because shape spaces are nonlinear and follow a quotient structure. Additionally, shape analysis often requires tools from functional data analysis (Ramsay and Silverman 2005) due to the infinite dimensionality of shape representation spaces. The goal of statistical shape analysis is to reproduce basic statistical techniques while taking into account these extra challenges. The developed techniques can then be used in real-life applications, most notably in medical imaging, where the shapes of anatomical structures can potentially be used to diagnose and monitor various types of diseases.

Many methods have been developed to analyze the shapes of objects observed in images. In the case of surfaces, one of the most popular approaches in medical imaging studies shapes of objects by embedding them in volumes and deforming the volumes (Beg et al. 2005; Grenander and Miller 1998; Joshi et al. 1997) (termed deformable templates and Large Deformation Diffeomorphic Metric Mapping or LDDMM). Others have studied 3D shape variability using level sets (Malladi et al. 1996), medial axes (Bouix et al. 2001; Gorczowski et al. 2010), or point clouds via the iterative closest point algorithm (Almhdie et al. 2007). The case of curves has also been considered under many different representations. In statistics, the most widely-used method was developed by Kendall (1984), where shapes were represented using a finite set of “important” points known as landmarks. The landmarks can be selected manually or automatically. Most often, they are provided by an application domain expert and correspond to similar salient features across a population of shapes; such landmarks are referred to as anatomical. On the other hand, mathematical landmarks correspond to points which in some sense capture the most important properties of the shapes (e.g., peaks and valleys). By reducing the representation of an object to a set of landmarks, one can alter multivariate statistical techniques to account for desired shape invariances, and use them for shape analysis. Developments of these techniques are described in many places (Bookstein 1986; Dryden and Mardia 1992, 1998; Small 1996).

The ability to apply classical multivariate analyses to landmark shapes is intriguing, but not without drawbacks. Landmark-based methods require the user to summarize the full outline of an object into a finite set of points. This leads to a loss of information, which may affect statistical conclusions. In addition, selecting landmarks is not a simple task; it is not clear how many points should be selected, or if there is an “optimal” configuration of points which best represents the objects’ outlines. One could select a large number of landmarks to better approximate the shape; however, this leads to a very high-dimensional problem, which can be quite challenging computationally. To overcome these challenges, several groups proposed methods that retain all information provided about the object’s outline. In this setting, one defines infinite-dimensional representations, which additionally requires invariance to re-parameterizations of the functions representing the curve or surface (in addition to the similarity group, which includes translation, rotation and scale). In the case of curves, parameterization determines how fast it is traversed. In the case of surfaces, parameterization defines its grid or mesh. Thus, changing the parameterization of curves or surfaces is a shape preserving transformation. In a statistical shape analysis context, re-parameterizing an object can be used to determine which geometric features of objects are in correspondence with each other.

Several authors have studied this new set of shape frameworks in-depth. Zahn and Roskies (1972) and Klassen et al. (2004) in the case of curves, and Brechbühler et al. (1995) and Styner et al. (2006) in the case of surfaces, achieve parameterization invariance through normalization (to arc-length for curves and equal area for surfaces). Unfortunately, these methods do not match geometric features of objects across a population of shapes, and thus result in suboptimal correspondences and subsequent statistical results (Kurtek et al. 2011b, 2012a,b; Srivastava et al. 2011). On the other hand, there is a set of methods in the statistics literature that seek “optimal” correspondences across a population of shapes; these methods are based on elastic metrics and are thus referred to as elastic in short. Instead of normalizing parameterizations, they seek a “best” re-parameterization to match one object to another. This process is often also referred to as registration. Such methods have been developed for statistical shape analysis of curves in Younes (1998), Younes et al. (2008), Joshi et al. (2007), Srivastava et al. (2011) and Kurtek et al. (2012b), and surfaces in Kurtek et al. (2010, 2011a,b, 2012a), Jermyn et al. (2012) and Samir et al. (2014). One of the main benefits of using elastic methods for shape analysis is that the metric used to calculate distances between shapes measures the amount of bending and stretching required to deform one object into another, thus providing a natural interpretation. However, elastic metrics in general are very difficult to work with due to their complex structure (Mio et al. 2007). Recent approaches developed elastic representations of curves and surfaces that greatly simplify the problem at hand (Jermyn et al. 2012; Kurtek et al. 2010; Srivastava et al. 2011).

In this manuscript, we describe an approach to statistical shape analysis that unifies the recent elastic method with previous landmark-based approaches. As mentioned earlier, relying on landmarks reduces the amount of information used in statistical analyses. However, while elastic shape analysis overcomes this problem, these methods treat all points as equally important. Thus, if special landmark locations (e.g., anatomical features) are known, standard elastic shape analysis methods are not able to emphasize these points. Thus, the ability to combine elastic shape analysis with landmark information allows us to overcome both drawbacks. As a motivating example, in medical imaging, an anatomical structure of interest is often represented as a surface. Additionally, the doctor marks special points on the structure, which can be used to detect abnormalities. These points (landmarks) are certainly valuable for statistical inferences, and thus including them in the analysis is necessary.

As another motivating example, consider the dog shapes shown in Fig. 1. In the left panel is a representation of the dog via a curve while in the right panel we show the outline of a dog as a surface. In both cases, it appears natural to place anatomical landmarks at the dogs’ legs, tail and snout. In the surface case, one can also clearly see the dog’s ears where additional landmarks can be marked. All of these points are important to representing the structure of the full dog outline, and should thus be incorporated into the shape analysis framework. Additionally, good landmark correspondences across shapes provide improved registration over unconstrained elastic methods as shown by Strait et al. (2017) and Kurtek et al. (2013a).

Fig. 1

Landmark- constrained curve and surface representations of a dog. Landmarks segment the full curve outline of the dog into six pieces. Landmarks are shown as black points on the curve or surface

The idea of incorporating landmark constraints into elastic representations of shape had not been explored much in the past. Liu et al. (2010) imposed soft landmark constraints on the analysis by augmenting the elastic shapes with an auxiliary function constructed using the landmark locations. Two recent papers provide statistical shape analysis tools for both (curves and surfaces) that are able to incorporate hard landmark constraints into elastic representations (Kurtek et al. 2013a; Strait et al. 2017). The methods presented in this manuscript are largely based on those works. The rest of this paper is organized as follows. In Sect. 2, we present tools for landmark-constrained registration and elastic comparison of shapes of curves and surfaces. Sect. 3 provides methods for averaging and summarization of variability of a sample of shapes. Throughout these two sections we illustrate the approach using multiple examples. Finally, we give a brief summary in Sect. 4.

2 Landmark-Constrained Shape Analysis

In this section, we describe a landmark-constrained elastic shape analysis framework for curves and surfaces. We begin by briefly discussing a technical issue that arises when using standard $\mathbb {L}^2$-based methods in this setting. We describe this issue for curves only, but note that it also arises in the same way for surfaces.

Let $\mathcal {F}$ represent an appropriate representation space of curves made precise later. Also, let Γ be the set of all diffeomorphisms of the curve domain. The set Γ contains all possible re-parameterizations of curves, and for an object $f\in \mathcal {F}$ and an element γ ∈ Γ, f ∘ γ represents its re-parameterization. Given this setup, many works in literature adopt the standard approach of measuring distances between elements of $\mathcal {F}$ using the $\mathbb {L}^2$ norm. Unfortunately, this framework is inappropriate for statistical shape analysis of parameterized curves as was previously shown in multiple places (Kurtek et al. 2010; Srivastava et al. 2011; Younes 1998). We elaborate next. Let $f_1,\ f_2\in \mathcal {F}$ be two parameterized curves, and γ ∈ Γ a re-parameterization function. Then, it is easy to show that the $\mathbb {L}^2$ norm is not preserved under the action of Γ, i.e., ∥f ₁ − f ₂∥≠ ∥f ₁ ∘ γ − f ₂ ∘ γ∥. Thus, in this setup, a common re-parameterization of two curves changes the distance between them (this is also termed “lack of isometry”). This theoretical problem prevents one from defining a parameterization-invariant statistical framework for shape analysis. Thus, in the following sections, we describe an approach which uses new representations of curves and surfaces that satisfy this property under the $\mathbb {L}^2$ metric. Furthermore, we show that one can seamlessly incorporate hard landmark constraints into these representations. For more detailed descriptions of these methods please refer to Srivastava et al. (2011), Strait et al. (2017), Jermyn et al. (2012) and Kurtek et al. (2013a).

2.1 Unconstrained Representation Spaces of Curves and Surfaces

Curves

Let $\mathcal {F}$ denote the space of two-dimensional, absolutely continuous curves with domain D = [0, 1] (planar open curves). The framework is also applicable to closed curves with domain $D=\mathbb {S}^1$ with minimal changes. Let $\varGamma =\{\gamma :[0,1]\to [0,1]|\gamma (0)=0,\ \gamma (1)=1,\ 0<\dot {\gamma }<\infty \}$ denote the unconstrained re-parameterization group, where $\dot {\gamma }$ is the derivative of γ. As stated earlier, this group does not act on $\mathcal {F}$ by isometries under the $\mathbb {L}^2$ metric. To circumvent this issue, for a curve $f\in \mathcal {F}$, we define the square-root velocity function (SRVF) representation of curves (Srivastava et al. 2011) as $q^{SRVF}(t)=\frac {\dot {f}(t)}{\sqrt {|\dot {f}(t)|}}$, where |⋅| is the Euclidean norm in $\mathbb {R}^2$. The inverse mapping is defined as $f(t) = f(0) + \int _0^t{q^{SRVF}(r)| q^{SRVF}(r) |} dr$; thus, the mapping from curve to SRVF is a bijection up to a translation. An important fact about the SRVF is that the action of Γ becomes $(q^{SRVF},\gamma )=(q^{SRVF}\circ \gamma )\sqrt {\dot {\gamma }}$. Note that the SRVF representation can be used for curves of any dimension, though the focus here is on two-dimensional curves.

Surfaces

In similar fashion, one can define a new representation of surfaces. In this case, let $\mathcal {F}$ represent the space of all smooth embeddings of $\mathbb {S}^2$ in $\mathbb {R}^3$ and let Γ be the set of all diffeomorphisms from $\mathbb {S}^2$ to itself. Again, Γ serves as the re-parameterization group for spherical surfaces. In this work, we only consider closed or spherical surfaces, but this framework is readily applicable to other types of surfaces including quadrilateral, hemispherical, cylindrical, etc. For a closed surface $f\in \mathcal {F}$, f ∘ γ represents its re-parameterization (i.e., the action of the re-parameterization group is the same as in the case of curves). To define a new representation of surfaces that allows parameterization-invariant shape analysis, let $n(s)=\frac {\partial f}{\partial u}(s) \times \frac {\partial f}{ \partial v}(s) \in \mathbb {R}^3$ denote the normal vector to the surface at the point $s=(u,v) \in \mathbb {S}^2$. Jermyn et al. (2012) defined the square-root normal field (SRNF) as $q^{SRNF}(s) = \frac {n(s)}{\sqrt {|n(s)|}}$, where |⋅| denotes the Euclidean norm in $\mathbb {R}^3$. If a surface f is re-parameterized to f ∘ γ, then its SRNF is given by $(q^{SRNF},\gamma )=(q^{SRNF}\circ \gamma )\sqrt {J_\gamma }$, where J _γ is the determinant of the Jacobian of γ. Note that unlike in the case of curves, inversion of SRNFs cannot be performed analytically. The numerical inversion of SRNFs has been considered by Xie et al. (2014) and Laga et al. (2017), and is a difficult computational problem. As defined here, SRNFs are only applicable to shape analysis of two-dimensional surfaces embedded in $\mathbb {R}^3$.

Advantages of SRVFs and SRNFs

There are two main benefits of these new mathematical representations of curves and surfaces: (1) the group Γ acts on the space of SRVFs or SRNFs by isometries under the $\mathbb {L}^2$ metric, and (2) the $\mathbb {L}^2$ metric on the space of SRVFs (SRNFs) corresponds to an elastic metric (partial elastic metric) on the original space of absolutely continuous curves (smooth surfaces). In both cases, the resulting representation space after transformation is $\mathbb {L}^2$, henceforth denoted by $\mathcal {Q}$. The relationship of these two representations to elastic metrics is an important property as they provide a measure of the amount of bending and stretching to deform one curve/surface into another. This allows natural interpretation of the shape distance between objects as well as natural shape deformation paths as will be seen in later sections. Further details on the elastic metric can be found in Srivastava et al. (2011), Mio et al. (2007) and Kurtek et al. (2012b). Note that whenever our discussion applies to either the SRVF or SRNF, we use q without the superscript to denote the representation.

Definition of the Pre-shape Space

Recall that shape is defined as a property of an object that is invariant to translation, scale, rotation and re-parameterization. The SRVF and SRNF representations, and associated elastic metrics, are automatically invariant to translation due to their definition through first derivatives only. For curves, scale invariance in this framework is achieved by re-scaling to unit length via $\int _0^1 |\dot {f}(t)| dt = \int _0^1 |q^{SRVF}(t)|{ }^2 dt = \|q^{SRVF}\|{ }^2 = 1$. For surfaces, we re-scale them to have unit area: $\int _{\mathbb {S}^2} |n(s)| ds = \int _{\mathbb {S}^2} |q^{SRNF}(s)|{ }^2 ds = \|q^{SRNF}\|{ }^2 = 1$. Thus, the resulting SRVFs or SRNFs lie on the unit Hilbert sphere, which forms the pre-shape space: $\mathcal {C}=\{q\in \mathcal {Q} | \|q\| = 1 \}$ (in the case of closed curves there is an additional closure condition). We refer to $\mathcal {C}$ as the pre-shape space because up to this point, we have only accounted for translation and scaling variabilities. Invariance to rotation and re-parameterization is obtained differently, using equivalence classes.

2.2 Landmark-Constrained Shape Space for Curves

We begin by introducing landmark constraints into the SRVF representation as they play an important role in the rotation and re-parameterization steps. Suppose that in addition to the curve f, we are given k discrete landmarks marked on f, $\{f(t_1),\dots ,f(t_k)\}\in \mathbb {R}^2$. Also, let SO(2) denote the group of all rotations in 2D (also called the special orthogonal group). To take into consideration the landmark constraints that were marked on the curve f, we must redefine the set of allowed re-parameterizations as a subgroup of the unconstrained re-parameterization group Γ whose elements respect landmark matching. For this purpose, we define $\varGamma _0=\{\gamma :[0,1]\to [0,1]|\gamma (0)=0,\ \gamma (1)=1,\ 0<\dot {\gamma }<\infty ,\ \gamma (t_i)=t_i,\ i=1,\dots ,k\}\subset \varGamma$as the landmark-constrained re-parameterization group. Applying two elements, O ∈ SO(2) and γ ∈ Γ ₀, to a curve f yields the transformed curve O(f ∘ γ), where the landmark points remain unmoved; the SRVF of this transformed curve is given by $O(q^{SRVF} \circ \gamma ) \sqrt {\dot {\gamma }}$. Then, the landmark-constrained shape space, denoted by $\mathcal {S}$, is defined by the set of equivalence classes $[q^{SRVF}] = \{O(q^{SRVF} \circ \gamma ) \sqrt {\dot {\gamma }} | O \in SO(2),\ \gamma \in \varGamma _0\}$ (note that these SRVF equivalence classes correspond to equivalence classes on the space of the original curves given by [f] = {O(f ∘ γ)|O ∈ SO(2), γ ∈ Γ ₀}). Thus, the landmark-constrained shape space is a quotient space: $\mathcal {S} = \mathcal {C} / (SO(2) \times \varGamma _0)$. The equivalence classes [q ^SRVF] represent the landmark-constrained shapes uniquely, and the shape space $\mathcal {S}$ provides the desired invariances to translation, scaling, rotation, and landmark-constrained re-parameterization.

Next, we define a suitable metric on $\mathcal {S}$. As mentioned earlier, an important property of the elastic metric on the space of absolutely continuous curves is that, under the SRVF representation, it is equivalent to the standard $\mathbb {L}^2$ metric (Srivastava et al. 2011). We begin by defining the elastic distance between two curves using their SRVF representation on $\mathcal {C}$. Since the pre-shape space $\mathcal {C}$ is a Hilbert sphere, the geodesic distance between two curves represented via their SRVFs $q_1,\ q_2 \in \mathcal {C}$ is given by $d_{\mathcal { C}}(q^{SRVF}_1,q^{SRVF}_2)=\theta = \cos ^{-1}(\langle q^{SRVF}_1,q^{SRVF}_2 \rangle )$, where 〈⋅, ⋅〉 is the $\mathbb {L}^2$ inner product; the corresponding geodesic path (optimal deformation path) is given analytically by $\alpha (q^{SRVF}_1,q^{SRVF}_2)(\tau )=\frac {1}{\sin {}(\theta )}(\sin {}((1-\tau )\theta )q^{SRVF}_1+\sin {}(\tau \theta )q^{SRVF}_2),\ \tau \in [0,1]$. The rotation and landmark-constrained re-parameterization groups act on $\mathcal {C}$ by isometries, which allows the $\mathbb {L}^2$ metric to descend from the pre-shape space to the quotient shape space. Then, the landmark-constrained geodesic distance in the shape space $\mathcal {S}$ is given by:

$$\displaystyle \begin{aligned} d([f_1],[f_2])\equiv d_{\mathcal{S}}([q^{SRVF}_1],[q^{SRVF}_2])= \min_{O \in SO(2),\ \gamma \in \varGamma_0} d_{\mathcal{C}}(q^{SRVF}_1,O(q^{SRVF}_2 \circ \gamma)\sqrt{\dot{\gamma}}).\end{aligned} $$

(1)

The optimization over SO(2) and Γ ₀ is often referred to as the registration process, which aligns geometric features across shapes. The optimal rotation is found using Procrustes analysis, which involves singular value decomposition (SVD). To optimize over Γ ₀, one can take a product space approach where the complete optimization problem is separated into an optimization over the unconstrained re-parameterization group Γ for each segment formed using the landmark constraints (see left panel of Fig. 1). See Strait et al. (2017) and Robinson (2012) for the implementation details. Once the optimal pair (O ^∗, γ ^∗) is found, one can compute the geodesic path in the SRVF shape space using $\alpha (q^{SRVF}_1,O^*(q^{SRVF}_2 \circ \gamma ^*) \sqrt {\dot {\gamma ^*}})$, and map it back to the space of absolutely continuous curves for visualization purposes. This procedure provides the landmark-constrained elastic geodesic path and distance between shapes of two curves.

2.3 Landmark-Constrained Shape Space for Surfaces

In contrast to the elastic curve framework, it is not a simple task to invert an arbitrary SRNF to obtain its original surface (Laga et al. 2017; Xie et al. 2014). Thus, in this case, we work directly in the space of smooth surfaces under the pullback of the $\mathbb {L}^2$ metric from the SRNF space; this is the previously mentioned partial elastic metric (Jermyn et al. 2012). We provide some details next.

Throughout this section, with a slight abuse of notation, we use $\mathcal {C}$ as the pre-shape space of smooth surfaces rather than their SRNF representations. Let each surface $f\in \mathcal {C}$ be annotated by k landmark points. Let s ₁, …, s _k be the locations of these landmarks on $\mathbb {S}^2$ such that $f(s_i)\in \mathbb {R}^3$, i = 1, …, k are the given landmarks on the parameterized surface f. To form the landmark constrained shape space we define $\varGamma _0 = \{\gamma :\mathbb {S}^2 \to \mathbb {S}^2 | \gamma \ \text{ is}\ \text{a}\ \text{diffeomorphism},\ \gamma (s_i) = s_i,\ i=1,\dots ,k\} \subset \varGamma$ as the landmark-constrained re-parameterization group for spherical surfaces. The rotation group SO(3) acts on the pre-shape space as (O, f) = Of; the constrained re-parameterization group Γ ₀ acts on $\mathcal {C}$ as before by composition (f, γ) = (f ∘ γ). Then, an equivalence class of a surface f is given by [f] = {O(f ∘ γ)|O ∈ SO(3), γ ∈ Γ ₀}, and represents a landmark-constrained shape of a spherical surface uniquely. The set of all such equivalence classes is defined to be the landmark-constrained elastic shape space denoted by $\mathcal {S}$. As in the case of curves, because SO(3) and Γ ₀ act on $\mathcal {C}$ by isometries, the partial elastic metric descends from the pre-shape space to the quotient space $\mathcal {S}$.

The shape geodesic between two landmark-constrained surfaces f ₁ and f ₂, such that f _j (s _i ), i = 1, …, k, and j = 1, 2, denote the landmarks on them, is defined as:

$$\displaystyle \begin{aligned} d_{\mathcal{S}}([f_1],[f_2])=\min_{O\in SO(3),\ \gamma\in\varGamma_0} \left( \min_{\begin{array}{c} F: [0,1] \to \mathcal{C} \\ F(0) = f_1,\ F(1) = O(f_2 \circ \gamma) \end{array} } \left( \int_0^1 \langle\langle \frac{dF}{dt}(t),\frac{dF}{dt}(t)\rangle\rangle ^{(1/2)}~dt \right) \right), \end{aligned} $$

(2)

where F(t) is a path in $\mathcal {C}$ and 〈〈⋅, ⋅〉〉 is the partial elastic metric. The quantity $L[F] = \int _0^1 \langle \langle \frac {dF}{dt}(t),\frac {dF}{dt}(t)\rangle \rangle ^{(1/2)}dt$ denotes the length of the path F. The inside minimization problem seeks the shortest path (geodesic) between f ₁ and O(f ₂ ∘ γ) in $\mathcal {C}$; the solution can be found using a path-straightening algorithm (Klassen and Srivastava 2006; Kurtek et al. 2012a; Samir et al. 2014). In the presented results, we approximate the geodesic using a straight line path. The outside minimization problem seeks an optimal landmark-constrained registration between f ₁ and f ₂, which is a search for an optimal rotation O ^∗∈ SO(3) and an optimal landmark-constrained re-parameterization γ ^∗∈ Γ ₀. The search for O ^∗ is again performed using Procrustes analysis. To find γ ^∗, we require two steps: (1) an initialization that matches given landmark points on f ₁ and f ₂, and (2) a gradient descent search over Γ ₀ that finds the optimal landmark-constrained re-parameterization. We begin with a description of the first step.

Initial Landmark Matching

First, we must find two initial diffeomorphisms $\gamma _1,\ \gamma _2:\mathbb {S}^2\to \mathbb {S}^2$ that map the selected landmarks on surfaces f ₁ and f ₂ with locations $\{\tilde {s}_1,\dots \tilde {s}_k\}\in \mathbb {S}^2$ and $\{\bar {s}_1,\dots \bar {s}_k\}\in \mathbb {S}^2$, respectively, to a standard set of landmarks ($s_1,\dots ,s_k\in \mathbb {S}^2$). We briefly describe this procedure for γ ₁. In general, the deformations between the landmarks can be very large. Thus, we divide the original problem into l smaller deformation steps. We first connect each pair of matched landmarks on $\mathbb {S}^2$ with a geodesic (great circle) and sample it uniformly using l steps. Then, we begin by solving for the first small deformation that matches the first point to the second point on this geodesic path for all landmarks. Let η _j be the tangent vector to $\mathbb {S}^2$ at $\tilde {s}_j$ such that $\exp _{\ \tilde {s}_j}(\eta _j) = s_j$; $\exp$ denotes the exponential map on the unit two-sphere. For brevity, we do not provide the full expression for the exponential map, but note that it is available analytically. Then, using a Gaussian kernel, we define a vector field over $\mathbb {S}^2$ according to $V(s) = \sum _{j=1}^k K(s, s_j) \eta _j$ in such a way that V (s _j ) = η _j . This is a simple interpolation step that uses the landmark vector fields to define a full vector field over $\mathbb {S}^2$. The desired small deformation at each point s is obtained by computing exp _s (V _s ). This is repeated for each of the l small steps. The desired large deformation γ ₁, which guarantees that the landmarks on f ₁ are matched exactly to the standard landmarks, is obtained through composition of the l small deformations. The procedure can be repeated for the second surface f ₂ in the same manner. This general procedure is also described in Kurtek et al. (2013a).

Gradient Descent Optimization Over Γ ₀

One can perform the optimization over Γ ₀ directly as suggested by Eq. (2). However, the correspondence between the partial elastic metric on the space of smooth closed surfaces and the $\mathbb {L}^2$ metric on the space of SRNFs allows us to greatly simplify the problem. Given two surfaces f ₁ and f ₂ with matched landmarks, the optimization over Γ ₀ is solved using the energy $E_{\gamma _{id}}(\gamma )=\|q^{SRNF}_1-\phi _{q^{SRNF}_2}(\gamma )\|{ }^2$, where $q^{SRNF}_1,\ q^{SRNF}_2$ are the SRNFs of f ₁ and f ₂ and $\phi _{q^{SRNF}_2}(\gamma )=(q^{SRNF}_2\circ \gamma )\sqrt {J_\gamma }$. An algorithm for finding optimal landmark-constrained re-parameterizations of surfaces via this energy is given in Kurtek et al. (2013a, 2010, 2011b) and is omitted here for brevity. We refer the interested reader to those papers for the details.

2.4 Motivating Examples

Landmark-Constrained Curves

The utility of landmark-constrained shape analysis for curves is presented in Fig. 2 for two complicated outlines of elephants. These elephants are fairly similar with two noticeable differences. First, the trunk of the first elephant is oriented downwards, but upwards for the second one. Also, the first elephant has four distinct legs (with very wide gaps in-between), while the second elephant has no gap between the second and third legs; visually, this feature looks like one large leg, although it is obvious that it should represent two legs. By identifying these special, salient features (i.e., the feet and trunk) as landmarks, we force them to be in correspondence. In some cases, this is a necessary constraint to enforce, as a standard unconstrained elastic shape analysis framework may not know how to handle the lack of a gap between the two legs on the second elephant or the very different orientations of the trunks.

Fig. 2

Top left: Outlines of two elephants used for shape comparison. The starting point (a landmark) is shown in red, while additional landmarks are in green. Top right: Correspondence of features between the two elephants and the optimal landmark-constrained re-parameterization function. Bottom: Landmark-constrained geodesic path between the two elephants, with landmarks marked along the path

The landmark-constrained geodesic distance between this pair of shapes is 0.7803, which provides an accurate measure of their shape differences; this claim is supported by the image displayed in the bottom panel of Fig. 2, which shows seven equally-spaced points along the shape space geodesic path where the initial shape is the first elephant and the final shape is the second elephant. This path shows how one can optimally deform the first elephant into the second, while preserving the landmarks (marked by points in the figure); it represents a natural deformation between the two given elephant shapes. The landmarks remain in correspondence throughout, and one can clearly see the expected shift in the orientation of the trunk as well as the reasonable transformation from being able to see all four legs distinctly to the gap disappearing between the middle two legs.

The top-right panel of Fig. 2 shows two additional plots associated with the landmark-constrained elastic matching obtained for the two elephants. The left plot shows the correspondence of points on the first elephant (red) to the points on the second elephant (blue); the landmark-constrained analysis ensured that geometric features around the trunks and legs match each other well. The right plot shows the optimal matching function (landmark-constrained diffeomorphism). Deviations from a straight, 45^∘ line indicate the elastic nature of the matching problem.

Landmark-Constrained Surfaces

Figure 3 displays a motivating example for the surface case. In the top left panel, we show two highly articulated surfaces of a standing cat and a standing horse. On top of each surface, we marked seven landmark points corresponding to natural features of the two animals (ears, legs and tail; the landmark on the tail of the horse is occluded). In the bottom panel, we show the initial landmark matching procedure. The leftmost spherical domain contains the two sets of landmarks as given on the surfaces (red for cat and black for horse). First, we compute an optimal rotation of the domain to match the two landmark sets as well as possible (this is an area-preserving element of Γ). The result is given on the middle sphere. Note that the landmarks are now closer than previously. Finally, we compute the nonlinear, large deformation that matches the two sets of landmarks exactly. For reference, we still show the original landmark locations. We also display an intermediate matching result in magenta. After initial landmark matching, we compute the landmark-constrained registration and geodesic between the two surfaces. The geodesic path is displayed in the top right panel of the figure. The path preserves important features of the two animal models; the two surfaces naturally deform into each other.

Fig. 3

Motivating example for landmark-constrained elastic shape analysis of spherical surfaces

2.5 Additional Examples

Figure 4 provides four additional examples of geodesic paths on the landmark-constrained shape space of curves. All examples in this manuscript were generated using the MPEG-7 dataset.¹ All of these examples were selected because of the potential for ambiguities in the matching of features using unconstrained elastic shape analysis (i.e., without landmarks). The first example compares two octopi, where the arms are in drastically different locations. Without the ability to constrain the comparison at the arm locations, the result may not display a natural path between the two shapes. However, placing eight meaningful landmarks allows for the geodesic path to show a natural movement of the octopus arms. Next to the octopus example is a comparison of two crowns, the first of which has five distinct tips, while the second one appears to have more than five tips. The landmark constraints in this case allow the extra tips to grow out of the gaps between tips in the first crown. The last two comparisons consider cows and butterflies, respectively. As in the previous examples, the addition of landmarks to the elastic representation provides valid deformations between shapes.

Fig. 4

Landmark-constrained geodesic paths for several examples from the MPEG-7 dataset. Corresponding landmarks are marked along each path

Next, we provide several examples of landmark-constrained geodesic paths (approximated using linear paths) between shapes of very complex spherical surfaces including dogs, cats, horses and human bodies. The models used in all of the examples in this manuscript were obtained from the TOSCA (Bronstein et al. 2008) and SHREC 2007 (Girogi et al. 2007) databases. The results are presented in Fig. 5. We do not show the marked landmarks in these cases, which were chosen as extreme points on the surfaces, i.e., legs, ears, tails, etc. In all examples, geometric features are nicely preserved along the geodesic paths. Furthermore, all of the deformations are natural: movement of arms and legs reflects our intuition. For example, the first path deforms a sitting dog into a laying dog. Note that at each point along the path the head of the dog is slightly lowered while the front limbs simply extend out. Figure 6 presents two examples where we compare landmark-constrained geodesics (top) to unconstrained ones (bottom). It is clear that, in both examples, landmarks add valuable information, which improves the comparison of the given shapes.

Fig. 5

Landmark-constrained geodesic paths for several examples from the TOSCA and SHREC 2007 datasets

Fig. 6

Comparison of two landmark-constrained elastic geodesic paths (top) to their unconstrained elastic counterparts (bottom)

3 Statistical Analysis of Landmark-Constrained Shapes

In this section, we provide two useful tools for statistical shape analysis of landmark-constrained curves and surfaces: computing the sample mean and summarizing variability using tangent principal component analysis (tPCA).

3.1 Sample Averaging

We begin by defining an intrinsic mean called the Karcher mean. Let {f ₁, f ₂, …, f _n } denote a sample of curves or surfaces. Then, the sample Karcher mean is given by $[\bar {f}] =\arg \min _{[f] \in \mathcal {S}} \sum _{i=1}^n d([f],[f_i])^2$. A gradient-based approach for finding the Karcher mean is given in Dryden and Mardia (1998) and Le (2001), and is omitted here for brevity; a specific implementation of this algorithm for SRVFs and SRNFs can be found in Kurtek et al. (2013b) and Kurtek et al. (2016), respectively. Further theoretical results and properties of Karcher means are given in Bhattacharya (2008), Bhattacharya and Bhattacharya (2012) and Bhattacharya and Lin (2017). Note that the resulting Karcher mean is defined as an entire equivalence class, which is how we defined shapes. For visualization purposes and subsequent covariance computation, we select one representative element $\bar {f}\in [\bar {f}]$. Next, we present several averaging results for landmark-constrained shapes of curves and closed surfaces.

3.1.1 Examples

We present two examples of Karcher averaging for a collection of landmark-constrained curves. Figure 7 shows the first example: a sample of 20 bones with different features. Some of the bones are slightly bent, and the edges of the bones vary from sharp to smooth. Four landmarks were selected on each bone. Two elastic averages are displayed; the middle average was computed without landmark constraints, while the right one includes landmarks. In this case, the two averages are somewhat similar; they both indeed look like bones, and it appears as if averaging over a somewhat large sample size has “smoothed” out any unusual features that a few of the bones may have. The difference in the two averages being fairly small suggests that landmark information may not be as crucial in this example.

Fig. 7

Left: Sample of 20 bone curves. Middle: Average of bones without landmark constraints. Right: Average of landmark-constrained bones (with landmarks annotated in black)

The second example features a sample of 12 camels displayed in Fig. 8; these camels vary in the number of clearly visible legs as well as the number of humps the animal has. Unconstrained Karcher averaging (without landmarks) does not preserve the legs well, as shown in the middle panel of Fig. 8. Thus, landmark-constrained Karcher averaging appears to be a better option to preserve as many features in the camels as possible. Six landmarks were selected on each camel (one for each of the four legs and one for each of the two possible humps). The landmark-constrained average camel shape is shown in the right panel of Fig. 8. Since all but two of the camels have two humps, the average also has two distinct humps. However, the Karcher mean’s legs show some signs of occlusion as the gap (particularly in the front pair of legs) is fairly small. This is inherited from the camels in the sample which share that property. For both examples, the gradient descent algorithm converged after approximately 500 iterations.

Fig. 8

Left: Sample of 12 camel outlines. Middle: Average of camels without landmark constraints. Right: Average of landmark-constrained camels (with landmarks annotated in black)

Finally, we close this section with one example of averaging shapes of landmark-constrained surfaces. The example is presented in Fig. 9 and considers a sample of eight horse shapes. The horses mostly differ in their pose. We selected eight landmarks on each horse corresponding to the two ears, the snout, the four legs and the tail. The resulting average is presented in the right panel. It is a nice representative of the given data where all features have been preserved. The pose of the average horse is approximately neutral.

Fig. 9

Left: Sample of eight horse surfaces. Right: Landmark-constrained average of the eight horse shapes

3.2 Summarization of Variability

Tangent principal component analysis (tPCA) is a useful way to visualize principal directions of variability in shape datasets. We first describe this procedure for landmark-constrained curves. Given the Karcher mean shape, we compute the shooting vectors v _i , i = 1, …, n by projecting all of the SRVFs into the linear tangent space at $\bar {q}$, the SRVF of $\bar {f}$, using the inverse exponential map. At the implementation stage, the shooting vectors are sampled using N points allowing us to use multivariate tools on this tangent space to perform tPCA. We first compute the sample covariance matrix given by $K=\frac {1}{n-1}\sum _{i=1}^nv_iv_i^T$ (assuming that the v _i s are stacked into long vectors). The SVD of K is given by K = UΣU ^T, where Σ is a diagonal matrix of principal component variances and the columns of U are the corresponding principal directions of variation in the data. One can explore the ith direction U _i by computing $\nu =t \sqrt {\varSigma _{ii}} U_i$ (for some value of t), where Σ _ii is the ith diagonal element of Σ. This vector can then be mapped back to the landmark-constrained shape space via the exponential map and converted to a curve for visualization. This procedure is greatly simplified by the invertibility of the SRVF. We can simply perform all of the analysis on the SRVF shape space (quotient space of the Hilbert sphere with a simple differential geometry), and then map the results back to the original curve shape space. This is not possible for the case of surfaces as described next.

The evaluation of the covariance for a collection of landmark-constrained surface shapes is performed as follows. First, we find the shooting vectors from the estimated Karcher mean $\bar {f}$ to each of the surfaces in the sample, $v_i = \frac {dF^*_i}{dt}|{ }_{t=0}$, where i = 1, …, n and F ^∗ denotes a geodesic path in the landmark-constrained shape space $\mathcal {S}$ (computed using path-straightening as before). To generate a much lower dimensional, orthonormal basis denoted by {B _j |j = 1, …, m}, m ≤ n, we apply the Gram-Schmidt procedure under the partial elastic metric 〈〈⋅, ⋅〉〉 to the observed shooting vectors {v _i , i = 1, …, n}. We approximately represent each original shape using a low dimensional coefficient vector c _i  = {c _i,j , j = 1, …, m}, where c _i,j  = 〈〈v _i , B _j 〉〉. The sample covariance matrix can be computed in the coefficient space as $K = \frac {1}{n-1} \sum _{i=1}^n c_i c_i^T \in \mathbb {R}^{m \times m}$ and tPCA can be performed using K as before. This results in the principal directions of variation in the given data U and the diagonal matrix of principal component variances Σ. To explore the principal direction $U_i \in \mathbb {R}^m$, we can compute the corresponding shooting vector as $\nu =t\sqrt {\varSigma _{ii}} \sum _{j=1}^m U_{i,j} B_j$ ($\sqrt {\varSigma _{ii}}$ denotes the ith diagonal element of Σ). One can then map this vector to a surface f using the exponential map. The exponential map in this case must be computed under the non-standard partial elastic metric introduced earlier, which is not a simple task. This can be accomplished using a tool called parallel transport, which was derived for this representation of surfaces by Xie et al. (2013). For brevity, we do not provide details here but rather refer the interested reader to that paper. In the current results, we approximate the exponential map using a straight line.

3.2.1 Examples

Like in many standard statistical analyses, one may want to understand the variability in a population given a collection of shapes. This can be done by looking at principal directions of variation obtained through tPCA. The top panel of Fig. 10 shows the top two principal directions of variation in the bone data. The primary direction includes a slight bending of the bone, as well as different patterns at the ends of the bone (especially at the top end). The second direction controls the thickness of the middle portion of the bone. Similarly, the middle panel of Fig. 10 displays the two principal directions of variation for the collection of camel shapes. The primary direction captures the variability in the presence of a gap between the front legs or the rear legs (or both) among the sample of shapes. The second one appears to capture some more differences in the leg structure as well as the variability in the humps. Finally, the bottom panel of Fig. 10 displays the same results for the sample of horse surfaces. The principal direction mainly reflects the up-down movement of the horse’s head and the pose of the front two legs. The second direction captures changes in the pose of the back legs and the overall body.

Fig. 10

Visualization of the two principal directions of variation for the bone, camel and horse examples. From left to right for each direction: −1.5, −1, −0.5, 0 (mean), +0.5, +1, +1.5 standard deviations from the mean

4 Summary

We present a framework for landmark-constrained shape analysis of curves and surfaces. The framework is based on elastic metrics and corresponding, simplifying representations termed the square-root velocity function and the square-root normal field. The elastic metrics combined with anatomical landmarks provide intuitive correspondences between shapes and result in natural geodesic deformations. We also provide tools for statistical analysis including averaging and summarization of variability using tangent principal component analysis. The resulting sample shape averages and principal directions of variability provide natural summaries of complex datasets.