Probabilistic Surface Reconstruction with Unknown Correspondence

Abstract

We frequently encounter the need to reconstruct the full 3D surface from a given part of a bone in areas such as orthopaedics and surgical planning. Once we establish correspondence between the partial surface and a Statistical Shape Model (SSM), the problem has an appealing solution. The most likely reconstruction, as well as the full posterior distribution of all possible surface completions, can be obtained in closed form with an SSM. In this paper, we argue that assuming known correspondence is unjustified for long bones. We show that this can lead to reconstructions, which greatly underestimate the uncertainty. Even worse, the ground truth solution is often deemed very unlikely under the posterior. Our main contribution is a method which allows us to estimate the posterior distribution of surfaces given partial surface information without making any assumptions about the correspondence. To this end, we use the Metropolis-Hastings algorithm to sample reconstructions with unknown pose and correspondence from the posterior distribution. We introduce a projection-proposal to propose shape and pose updates to the Markov-Chain, which lets us explore the posterior distribution much more efficiently than a standard random-walk proposal. We use less than 1% of the samples needed by a random-walk to explore the posterior. We compare our method with the analytically computed posterior distribution, which assumes fixed correspondence. The comparison shows that our method leads to much more realistic posterior estimates when only small fragments of the bones are available.

1 Introduction

Surface reconstruction is encountered in many different areas. The reconstructed surface can be used to guide the design of patient-specific implants in the medical area, or estimate the sex and ethnicity of an individual in forensic investigations [10, 11]. When only incomplete data is available, SSMs can be used to determine the most likely complete surface [2, 12, 13]. Nevertheless, the reconstruction becomes wrong and overconfident if correspondence cannot be obtained. In cases where the surfaces are free of distinctive features, such as e.g. the shaft of a femur bone, there might even be multiple equally likely reconstructions with different lengths. In medical applications, a certainty estimate for a reconstruction is often required. This estimate can be computed as the likelihood of the chosen reconstruction in the distribution of all other possible reconstructions. Such a full distribution of surfaces given partial knowledge about the solution is known in the Bayesian setting as the posterior distribution. Since an SSM is formulated as a distribution over shapes, it is possible to derive a posterior model if only part of a surface is given [1] or if knowledge such as weight, sex, or age of a patient is known [3]. Current methods compute the posterior distribution analytically by assuming both fixed pose and fixed point-to-point correspondence [1, 3]. Furthermore, the analytical-posterior requires an initial dataset alignment before it can be computed. In absence of exact point-to-point correspondence, those conditions are impossible to fulfil.

Fig. 1.

A femur SSM is first registered to the partial surface to establish point-to-point correspondence. If we assume fixed correspondence, the posterior is computed analytically. With our method, we assume unknown correspondence and compute the posterior as a Markov-Chain. Note the correspondence difference to the available data visualised with the yellow markers on the long and short bone reconstructions. The coloured bones show the uncertainty, computed as the sum of the variances at each landmark with the different methods. Our method clearly shows more variability (red) far away from the partial surface, but at the same time has low variability (blue) at the known area. (Color figure online)

We present a method to estimate the posterior distribution from partial surface knowledge. A similar method has previously been used for fitting an active shape model to a target image [9] to compute the most likely solution. In contrast, we compute the full distribution of possible reconstructions.

In Fig. 1, we show how our method compares to the analytical method. We use the Metropolis-Hastings (MH) algorithm to compute the Markov-Chain posterior distribution. We will be referring to our method as the sampling-posterior.

The default random-walk in MH takes a long time to converge. As the SSM should stay fixed around the given part of the surface, we have to use very small shape and pose updates. Informed sampling approaches overcome this problem by including knowledge from the current state into its proposal [4, 7]. We introduce a new projection-proposal, which keeps the known part of the model fixed and only varies the pose and shape in the unknown part. In [8], an Iterative Closest Point (ICP)-like proposal is introduced for surface registration, whereas our projection-proposal explores the space of possible surface reconstructions of partial surfaces and includes the variability in pose difference.

We apply the sampling-posterior to estimate the posterior distribution of partial femurs. The femur bone is used as an example as the full shape of the femur (as well as other long bones) is inherently difficult to estimate. Thus, there is almost no correlation between the shape of the upper and lower part and the length. We show the limitations of the current method by comparing our sampling-posterior to the analytical-posterior distributions of different partial femur bones. This paper makes the following contributions:

• We show how to compute the estimated posterior distribution of a partial surface without assuming a fixed point-to-point correspondence in Sect. 3.1.

• We introduce a new MH proposal to create independent samples and thereby speed up the posterior estimation process in Sect. 3.2.

• We show the limitations of the current analytical-posterior [1] in Sect. 2.1 and based on experimental results in Sect. 4.

2 Statistical Shape Models

SSMs learn the shape variability from examples shapes. When working with a shape $M_{i}$, we usually work with the vector representation $\varvec{s}_i=(p^{i}_{1_x}, p^{i}_{1_y}, p^{i}_{1_z} ,\ldots , p^{i}_{n_x},p^{i}_{n_y},p^{i}_{n_z})$ where $p\in \mathbb {R}^{3}$ is a landmark and n is the number of landmarks in the shape. A compact representation can be found by performing a Principal Component Analysis (PCA). The covariance matrix can then be represented by using $N-1$ basis functions. In matrix format, the shapes are represented as $\varvec{s}=\varvec{\mu }+UD\varvec{\alpha }=\varvec{\mu }+Q\varvec{\alpha }$, with $\varvec{\mu }$ being the mean shape, U being the matrix containing all the eigenvectors and D containing the square-root of the eigenvalues of the covariance matrix $\varSigma$. Each shape $M_{i}$ can then be determined by an $\varvec{\alpha }$ vector. The pose of the model can also be changed with both a translation vector $\varvec{t}=(t_{x}, t_{y}, t_{z})^{T}\in \mathbb {R}^{3}$ and a rotation matrix parameterised by the Euler angles $R(\phi , \psi , \rho )\in SO(3).$ All parameters are concatenated into one vector $\varvec{\theta }=(\alpha _{0},\ldots ,\alpha _{N-1}, \phi , \psi , \rho , t_{x}, t_{y}, t_{z})^{T}$ and we use the notation $M[\varvec{\theta }]$ to refer to the triangulated surface M defined by the parameter vector $\varvec{\theta }$. The scale is directly incorporated in our construction of the SSM, as it would otherwise be difficult to obtain a correct statistical size measures if the size of the SSM can be scaled arbitrarily.

2.1 Analytical Posterior Models

We compare the sampling-posterior method to the analytical-posterior described in [1]. The given part of a shape is described as $\varvec{s}_{g} \in \mathbb {R}^{3q}$ with q being the number of landmarks. In our model, this becomes $\varvec{s}_{g}=\varvec{\mu }_{g}+Q_{g}\varvec{\alpha }+I_{3q}\epsilon$ with $\epsilon \sim \mathcal {N}(0,\sigma ^{2})$ being the noise term of each landmark observation. The difficulty with the analytical-posterior is that point-to-point correspondence needs to be obtained before the $\varvec{s}_{g}$ vector can be defined. Furthermore, the rigid alignment needs to be fixed, resulting in the posterior distribution only containing shape variance.

In [1], the authors mention that all training shapes have to be aligned with respect to the subset of points available in $\varvec{s}_{g}$ in order to have a meaningful result. In the following, we will refer to the analytical-posterior computed without aligning according to the $\varvec{s}_{g}$ dataset as the naive-posterior and the analytical-posterior with the dataset alignment as the aligned-posterior.

3 Method

Now we explain how to compute the posterior distribution without assuming a fixed correspondence between the given data and the SSM. We define a probabilistic model over possible surface reconstructions (determined by $\varvec{\theta }$) given partial surface information $\varvec{s}_{g}$,

$$\begin{aligned} P(\varvec{\theta }|\varvec{s}_{g}) = \frac{P(\varvec{\theta })P(\varvec{s}_{g}|\varvec{\theta })}{\int P(\varvec{\theta })P(\varvec{s}_{g}|\varvec{\theta })d\varvec{\theta }}. \end{aligned}$$

(1)

The shape prior $P(\varvec{\theta })$ penalises unlikely shapes. The combined likelihood over all the points in the given surface $\varvec{s}_{g}$ is

$$\begin{aligned} P(\varvec{s}_{g}|\varvec{\theta })=\prod _{i=1}^{q}\mathcal {N}(d_{i}(\varvec{\theta }, \varvec{s}_{g}); 0,\sigma ^2), \end{aligned}$$

(2)

where $d_{i}$ is the Euclidean distance between the point $p_{i}$ in the partial surface $\varvec{s}_{g}$ to the closest point on the surface of $M[\varvec{\theta }]$. A similar likelihood function was used in [9] to measure the distance between an SSM and expert annotation in images. We define $\sigma ^2=1.0\,\text {mm}^2$ and the same for $\epsilon$ in the analytical-posterior in order for the posterior distributions to be comparable. Note that the distance likelihood assumes that no pathologies exist in the partial surface.

3.1 Approximating the Probabilistic Model

Unfortunately, the full distribution of surfaces given the partial surface, as in Eq. (2), cannot be obtained analytically. Instead, it is possible to compute the unnormalised density for any surface described with $\varvec{\theta }$. This allows us to use the MH algorithm to estimate the full posterior distribution. We use a random-walk to explore the shape space and have independent proposals for the translation $Q(\varvec{t}'|\varvec{t})$, rotation $Q(\varvec{R}'|\varvec{R})$, and shape $Q(\varvec{\alpha }'|\varvec{\alpha })$ parameters. As scaling is directly incorporated in our SSM, a scaling proposal is not used.

Fig. 2.

Visualisation of the projection-proposal steps as described in Sect. 3.2. (Color figure online)

3.2 Projection-Proposal

Ideally, we would like to keep the known part of the shape model fixed around $\varvec{s}_{g}$, as we are interested in the posterior distribution given partial surface information. With the random-walk proposals, we have to use very small shape and pose update steps. As a consequence many samples need to be taken before independent samples are found. Therefore, we suggest a projection-proposal to keep the shape at the known part of the model fixed and only vary the unknown part. The projection-proposal makes use of the analytical-posterior as described in Sect. 2.1. Before computing the analytical-posterior, we make a random rotation or translation proposal and compute the posterior distribution based on the initial position of the model. When computing the analytical-posterior, an anisotropic noise term $\epsilon$ is used. To simulate correspondence shift along the surface, we model it as a multivariate normal distribution with a low variance along the normal and a higher variance along the surface. The variance at each point $p_{k}$ in $M[\varvec{\theta }]$ is computed by

$$\begin{aligned} \varSigma _{p_{k}} = [\varvec{n},\varvec{v}_{1}, \varvec{v}_{2}] \begin{bmatrix}\sigma _{n}^{2} &{} 0 &{} 0 \\ 0 &{} \sigma _{v}^{2} &{} 0 \\ 0 &{} 0 &{} \sigma _{v}^{2} \end{bmatrix} [\varvec{n},\varvec{v}_{1}, \varvec{v}_{2}]^{T} \end{aligned}$$

(3)

where $\varvec{n}$ is the normal vector at the vertex $\varvec{p}_{k}$ in the surface and $\varvec{v}_{1}$ and $\varvec{v}_{2}$ are perpendicular vectors to the normal. The variance along each vectors is set as $\sigma _{n}^{2}=0.1\,\text {mm}^2$ and $\sigma _{v}^{2}=5.0\,\text {mm}^2$. The projection-proposal can be described in 5 steps with Fig. 2 as visualisation for each step:

• 1. Compute corresponding points by taking the closest points from the partial surface (red) $\varvec{s}_{g}$ to the current surface $M[\varvec{\theta }]$ (black). We compute $\varvec{s}_{g*}$ as the points in the SSM corresponding to the partial surface.

• 2. Propose a random pose update from $Q(\varvec{t}'|\varvec{t})+Q(\varvec{R}'|\varvec{R})$, while keeping the current shape parameters $\varvec{\alpha }$ fixed, such that a new $\varvec{\theta }'$ is computed ($M[\varvec{\theta }']$ shown in blue).

• 3. Compute the analytical-posterior $p(\varvec{\alpha }|\varvec{\theta }',\varvec{s}_{g*})$ based on Sect. 2.1.

• 4. Draw a sample from the distribution $p(\varvec{\alpha }|\varvec{\theta }',\varvec{s}_{g*})$ by randomly setting the $\varvec{\alpha }$ vector in the SSM (green shows the posterior mean).

• 5. Compute the $\varvec{\theta }''$ update based on the full SSM $p(\varvec{\alpha })$ (the proposed sample $M[\varvec{\theta }'']$ is shown in green).

Unlike the random-walk proposal, this proposal is not symmetric. Therefore, to ensure convergence of the MH algorithm, we need to be able to compute the transition probabilities of going from $\varvec{\theta }$ to $\varvec{\theta }'$ as well as from $\varvec{\theta }'$ to $\varvec{\theta }$ [6]. The transition probability can be computed using the posterior distribution from step 3 as also shown in [8].

Projection-Proposal Importance. We need i.i.d. samples to compute the variance, which means that we need to find the number of samples to be discarded from the Markov-Chain before an independent sample is found. We compute the autocorrelation of the individual shape parameters and look for the number of samples needed to reach 0 correlation. We report 50 samples for the projection-proposal and $30\times 10^{3}$ samples for the random-walk. While the random-walk requires 600 times more samples, the projection-proposal is only 6 times slower, making it overall 100 times faster.

We compute the bone length variance based on the distance between two landmarks. The length variation converges at $10^{3}$ samples with the projection-proposal. For the random-walk, we need $500\times 10^{3}$ samples to achieve the same length variance. With the projection-proposal, we can, therefore, explore similar variance numbers with less than $1\%$ of the samples needed by the random-walk.

Fig. 3.

Violin and box plots of bone length prediction in mm using the analytical-posterior and our sampling-posterior. All plots (a, b, c, d) concern the same ground-truth bone (length visualised with the red-dashed line), but differ in how much of the bone is given. The average Euclidean (E) and Hausdorff (H) distances from the ground-truth surface to the mean surface from the distributions are in mm. (Color figure online)

4 Evaluation

For the experiments, we use 61 femur meshes extracted from Computed Tomography (CT) images. We use 50 femurs for the femur SSM (femur lengths, mean: $372\,\text {mm}$, min: $322\,\text {mm}$, max: $437\,\text {mm}$) and 11 for the test-set (femur lengths, mean: $372\,\text {mm}$, min: $322\,\text {mm}$, max: $441\,\text {mm}$). The SSM contains a total of 1622 landmarks. Each test femur is divided into several partial meshes from where the posteriors are estimated. In Fig. 3, a subset of the cuts are shown¹.

Experimental Setup. We compare the sampling-posterior with the naive-posterior and the aligned-posterior. For the aligned-posterior, we need to estimate the observed points in the SSM. This is the same procedure that was done in step 1 of the projection-proposal. We perform a registration with the SSM and take the closest points to it from $\varvec{s}_{g}$. For the registration, we use the method from [9]. Alternatively, the non-rigid ICP algorithm can be used [5].

In the overview image of our method (Fig. 1), the posterior variability of the different methods is visualised with colours on the full femur bone. Very little variance is maintained in the naive-posterior, which highlights the importance of dataset alignment when computing the analytical-posterior. The sampling-posterior contains 2 to 3 times more variability than the aligned-posterior, suggesting that the full variability cannot be obtained using a fixed correspondence.

Length Estimation of Partial Bones. We compare the mean and the variance of bone lengths from the different posterior estimation methods. A landmark is placed at each end of the femur bones and the variability of the distance between the two landmarks is reported. For the analytical-posteriors, we randomly sample $10^{3}$ shapes from the posterior models to be used for the estimate. For the projection-proposal we take $10^{3}$ samples with 50 sample spacing in between. The bone length results for test femur 1 are shown in Fig. 3. Notice the difference between the results for partial bone a and c. More data is available in c, which results in a more narrow distribution, whereas the correspondence used in c is worse, making the ground-truth surface very unlikely under its distribution. The sampling-posterior results of the remaining test femurs are shown in Fig. 5.

We observe that both of the analytical-posterior methods sometimes fail to estimate the ground-truth length within their variability for most of the cuts. In contrast, the sampling-posterior can explain the shape length accurately.

Fig. 4.

Violin and box plots of bone length prediction in mm. All plots concern the same ground-truth bone (length visualised with the red-dashed line). The posteriors are computed with the aligned-analytical-posterior method and differ only in the correspondence which has been used. The average Euclidean (E) and Hausdorff (H) distances from the ground-truth surface to the mean surface from the distributions are in mm. (a) and (b) refers to the same partial shapes as in Fig. 3. (Color figure online)

Importance of Correct Correspondence. The quality of a surface reconstruction can be measured with the average Euclidean or Hausdorff distance to the ground-truth. These measures are, however, not a good indicator for the registration quality when large uncertainty exists in the correspondence. In Fig. 4 we show the same bone length experiment as in Fig. 3, but only for the aligned-posterior computed using different correspondences. The different correspondences have been found by initialising the SSM either as a very short, medium or long bone. We see that a close to perfect reconstruction can be found if the ground-truth correspondence is known, but at the same time can extremely over or underestimate the bone length if wrong correspondences is used. The average Euclidean distance from the partial surface to the reconstruction is, in all the cases, less than 0.25 mm, which suggests that the model represents the surface well in the available part.

Fig. 5.

Violin and box plots of bone length prediction in mm. All plots of different ground-truth bones. The posteriors are computed with our sampling method.

5 Conclusion

It is difficult to infer the full shape from a bone fragment. This is due to missing exact point-to-point correspondence. Previous methods deterministically find a set of correspondences to estimate the posterior. This can results in overconfident posterior estimates if incorrect correspondences are used. We have shown how previous methods even fail to explain the ground truth solution in an experimental setup with synthetic data. Our main contribution is a sampling approach that estimates the posterior distribution without relying on a fixed set of correspondences. We use the MH algorithm to obtain the variability in shape and pose reconstruction of partial surfaces. We have shown that the sampling-posterior, in contrast to the analytical methods, robustly is able to explain ground-truth data under its posterior. We also presented a technical contribution to the sampling-posterior in the form of a projection-proposal. This proposal is able to explore the posterior distribution more efficiently. With our sampling-posterior approach, both correspondence and reconstruction estimates are more accurate than with the traditional analytical approach. We are also able to more reliably estimate the uncertainty of the reconstruction results.