Keywords

9.1 Introduction

In order to reduce radiation exposure to patients, 2D-3D reconstruction, which can reconstruct 3D patient-specific models from 2D X-ray images, is proposed as an alternative to CT scan for certain applications. Depending on the output, those 2D-3D reconstruction methods can be generally classified into two categories [1]: 3D surface model reconstruction [2, 3] and 3D intensity volume reconstruction [4,5,6,7]. The methods in the former category compute 3D patient-specific surface models from one or multiple 2D X-ray images. No intensity information or information about cortical bone is available. The methods in the second category generate 3D patient-specific volumes from a limited number of X-ray images. Most of the previous work tried to solve the ill-posed problem of 2D-3D volume reconstruction by introducing different statistical prior models, while Yu et al. [7] firstly explored the potential of atlas-based 2D-3D intensity volume reconstruction. To our knowledge, none of the above mentioned methods have been applied to reconstruct the intensity volumes of a complete lower extremity.

In this chapter we present an atlas-based 2D-3D intensity volume reconstruction approach which is an extension of the previous work [7], and we apply it to reconstruct 3D intensity volumes of femur, tibia, and three muscles around the thigh region from a pair of 2D X-ray images.

The remainder of the chapter is arranged as follows: the techniques of the proposed atlas-based 2D-3D reconstruction method will be described in Sect. 9.2. Section 9.3 will present the results of our validation experiments on several datasets, followed by the discussions and conclusions in Sect. 9.4.

9.2 Materials and Methods

9.2.1 Atlas Preparation

The atlas consists of the template volumes of femur and tibia as well as the template volumes of rectus femoris muscle, vastus lateralis and intermedius muscle, and vastus medialis muscle (if reconstructing these thigh muscles) which are segmented from the CT data of a healthy lower extremity. In addition, the atlas includes two sets of sparse 3D landmarks \(\left (\left \{L_{\mathrm{femur},n}\right \}_{n=1}^{N_{1}}\right.\) and \(\left.\left \{L_{\mathrm{tibia},n}\right \}_{n=1}^{N_{2}}\right )\) extracted from the outer surfaces and the intramedullary canal surfaces of the template volumes.

9.2.2 The 2D-3D Reconstruction Pipeline

The 2D-3D reconstruction process is aiming to fit the atlas to a pair of X-ray images, one acquired from the anterior-posterior (AP) direction and the other from an oblique view (not necessary the lateral-medial (LM) direction). Both images are calibrated and co-registered to a common coordinate system called c. A template volume \(I\left (x\right )\) is aligned to the reference space c via a forward mapping: \(I\left (x_{c}\left (T_{g},T_{d}\right )\right ) = I\left (T_{g} \circ T_{d} \circ x_{f}\right )\), where x f is a point in the template space. Here, a global scaled-rigid transformation T g and a local deformation T d are to be determined via a 2D-3D scaled-rigid registration stage and a 2D-3D intensity volume reconstruction stage. Both stages are based on the procedure of 2D-2D nonrigid registration-based 3D landmark reconstruction.

The 2D-2D nonrigid registration-based 3D landmark reconstruction follows the previous work [7] which is organized in a hierarchical style: (1) DRR generation and 3D landmark projection, (2) nonrigid 2D-2D intensity-based registration, and (3) triangulation-based landmark reconstruction. Inspired by the work [2], 3D sparse landmarks instead of the B-spline control points used in the previous work are adapted.

Given the initial transformation of the template volumes to the common coordinate system c via landmark-based alignment, we can generate virtual 2D radiographic images and also project those 3D sparse landmarks. The nonrigid 2D deformation fields obtained from the registration module based on the registration library “elastix” [8] enable us to look for the dimensional correspondences represented by the paired 2D projected landmarks, and then new 3D sparse landmarks are reconstructed via triangulation.

9.2.2.1 2D-3D Scaled-Rigid Alignment

2D-3D scaled-rigid alignment is conducted via the paired-point matching between the reconstructed 3D landmarks and the original 3D landmarks in the atlas (see Fig. 9.1, left), and we iteratively compute \(\left \{T_{g}^{t}\right \}_{t=1,2,3,\ldots }\) in order to handle those complicated pose differences. The 2D-3D similarity alignment is applied to femur and tibia individually, and finally we can obtain two scaled-rigid transformations T g femur and T g tibia. Figure 9.1, right, shows an example of the 2D-3D scaled-rigid alignment which can handle large pose difference.

Fig. 9.1
figure 1

An illustration of 2D-3D scaled-rigid alignment

Full size image

9.2.2.2 2D-3D Intensity Volume Reconstruction

2D-3D intensity volume reconstruction starts with the reconstructed 3D sparse landmarks \(\left (\left \{L'_{\mathrm{femur},n}\right \}_{n=1}^{N_{1}}\right.\) or \(\left.\left \{L'_{\mathrm{tibia},n}\right \}_{n=1}^{N_{2}}\right )\) and the original 3D landmarks in the atlas \(\left (\left \{L_{\mathrm{femur},n}\right \}_{n=1}^{N_{1}}\right.\) or \(\left.\left \{L_{\mathrm{tibia},n}\right \}_{n=1}^{N_{2}}\right )\). Firstly, we transform these reconstructed landmarks back to the space of the atlas with T g −1,femur and T g −1,tibia, and then two local deformations T l femur and T l tibia can be computed using 3D thin-plate-spline transformation as follows:

$$\displaystyle\begin{array}{rcl} \left \{\begin{array}{*{20}{c}} T_{l}^{\mathrm{femur}} \leftarrow T_{TPS}\left (\left \{L_{\mathrm{femur},n}\right \}_{n=1}^{N_{1}},\left \{T_{g}^{-1,\mathrm{femur}} \circ L'_{\mathrm{femur},n}\right \}_{n=1}^{N_{1}}\right ) \\ T_{l}^{\mathrm{tibia}} \leftarrow T_{TPS}\left (\left \{L_{\mathrm{tibia},n}\right \}_{n=1}^{N_{2}},\left \{T_{g}^{-1,\mathrm{tibia}} \circ L'_{\mathrm{tibia},n}\right \}_{n=1}^{N_{2}}\right )\end{array} \right.& &{}\end{array}$$
(9.1)

Notice that the obtained transformations T l femur and T l tibia are usually ill-posed since there is no restriction on the behaviors of 3D deformation fields, which may lead to poor reconstruction results (see Fig. 9.2). Therefore, we apply an adaptive regularization on the B-spline grid(s) sampled from T l femur and T l tibia in order to derive the anatomically correct results.

Fig. 9.2
figure 2

A comparison of the reconstruction of femur and tibia without the adaptive regularization strategy

Full size image

The adaptive regularization strategy begins with the layout of a combined B-spline grid or two individual B-spline grids in terms of the articulated or individual 2D-3D reconstruction methods by interpolating the displacements at the control points from T l femur and T l tibia. Following the previous work [9], the displacement vectors \(\mathbf{d}_{ijk}\) at the control points are regularized based on the Neumann boundary condition on the control points [10]. Figure 9.3 illustrates the 3D deformations computed from each step of the regularization.

Fig. 9.3
figure 3

A comparison of the deformation fields derived from a TPS transformation and from the regularized B-spline transformation. (a) Voxel-wise 3D deformation field computed from the TPS transformation. (b) Displacements of the B-spline control points interpolated from the TPS transformation. (c) Adaptive regularization on the displacements of these B-spline control points. (d) Voxel-wise 3D deformation field interpolated from the regularized B-spline transformation

Full size image

In order to prevent the reconstructed femur and tibia from penetrating each other, we investigated two strategies to reconstruct the associated structures:

  1. I.

    Individual 2D-3D reconstruction. The reconstruction of femur and tibia is completely individual, and each time just one anatomy will be reconstructed. As indicated, there is no consideration over the articulation of the knee joint.

  2. II.

    Articulated 2D-3D reconstruction. A combined B-spline grid is placed over the space of the template volumes, and the displacement \(\mathbf{d}_{ijk}\) at a control point C ijk is computed either by T l femur or by T l tibia, depending on the relative position between a predefined axis-aligned plane ζ (see Fig. 9.4, left) and C ijk:

    Fig. 9.4
    figure 4

    The schematic view of the articulated 2D-3D reconstruction method (left) and the qualitative comparison with the individual 2D-3D reconstruction method (right)

    Full size image
    $$\displaystyle\begin{array}{rcl} \left \{\begin{array}{*{20}{c}} if\,\,\mathbf{C}_{ijk}\,\,is\,\mathrm{above}\,\zeta,\,\mathbf{d}_{ijk} \leftarrow T_{l}^{\mathrm{femur}}\left (\mathbf{C}_{ijk}\right ) \\ if\,\,\mathbf{C}_{ijk}\,\,is\,\mathrm{below}\,\zeta,\,\mathbf{d}_{ijk} \leftarrow T_{l}^{\mathrm{tibia}}\left (\mathbf{C}_{ijk}\right )\end{array} \right.& & {}\end{array}$$
    (9.2)

We found that the reconstruction accuracy is basically the same for both strategies, while the qualitative comparison of reconstructing knee joint structure demonstrates the superiority of the articulated 2D-3D reconstruction method over the individual one (see Fig. 9.4, right).

9.2.2.3 The Reconstruction of Three Muscles Around the Thigh Region

The obtained 3D deformation fields from the reconstruction pipeline provide the potential of reconstructing the muscles in the thigh region. Currently, we just focus on the reconstruction of (1) rectus femoris muscle, (2) vastus lateralis and intermedius muscle, and (3) vastus medialis muscle.

9.3 Experiments and Results

Approved by a local institution review board (IRB), we conducted three experiments to validate the proposed reconstruction pipeline in regard to different motivations.

9.3.1 Experiment on CT Dataset of 11 Cadaveric Legs

Each CT data has a voxel spacing of 0. 78 × 0. 78 × 1 mm, and we chose a healthy CT data from them to create the atlas for all experiments. For the atlas, we segmented the binary labels of the femoral and tibial structures as well as their cortical bone regions, and 641 landmarks for femur and 872 landmarks for tibia were extracted from these binary labels.

In this experiment, we would like to evaluate the overall reconstruction accuracy of femur and tibia as well as the reconstruction accuracy of their intramedullary canal regions. Therefore, for the left 10 sets of CT volumes, we segmented the binary labels of the femoral and tibial structures as well as their cortical bone regions for each CT data as the ground truth, and also we generated a pair of virtual 2D radiographic images (DRRs) as the reference images (see Fig. 9.5, top).

Fig. 9.5
figure 5

The reference images and the ground truth from each CT data (top) and the quantitative (bottom, left) and qualitative (bottom, right) results of the experiment conducted on ten cadaveric legs

Full size image

We assessed both the individual and the articulated 2D-3D reconstruction strategies, and the results are shown in Fig. 9.5, left. Here, the average surface distance (ASD) and the dice coefficient (DC) for the overall reconstruction and the reconstruction of cortical bone region (i.e., CBRASD and CBRDC) were measured. From the results, there is no statistically significant difference in accuracy, but it is distinct in the preservation of knee joint structure from the two strategies (see Fig. 9.4). Figure 9.5, right, shows a qualitative comparison of the reconstructed volumes with the associated ground truth volumes for both femur and tibia.

9.3.2 Experiment on X-Ray Images from Patients

Ten pairs of X-ray images were collected for this experiment, which is more challenging due to the image quality. Since only the CT data around three local regions (hip, knee, and ankle joint) were available, the reconstruction accuracy was evaluated by comparing the surface models extracted from the ground truth CT data with those extracted from the reconstructed volumes after rigidly aligning them together.

The average surface distance for the local regions including proximal femur (PF-ASD), distal femur (DF-ASD), proximal tibia (PT-ASD), and distal tibia (DT-ASD) were measured. The quantitative results are shown in Fig. 9.6, left, where an overall reconstruction accuracy of 1.4 mm was found, and Fig. 9.6, right, shows a reconstruction case.

Fig. 9.6
figure 6

The average surface distances measured between the reconstructed surface models and the ground truth surface models

Full size image

9.3.3 Experiment on Reconstructing Three Thigh Muscles

We also evaluated the accuracy of reconstructing three thigh muscles on a set of 12 one-side CT data with the associated ground truth segmentations around the thigh region [11]. One CT volume was randomly chosen to create the atlas, and we conducted the experiment on the left 11 cases. We measured the dice coefficient (DC) to evaluate the reconstruction accuracy of the three thigh muscles, and the results are shown in Fig. 9.7, ranging from 78% to 85%.

Fig. 9.7
figure 7

(a) The femur and thigh muscle reconstruction accuracy: red (ground truth surface) and green (reconstructed surface); (b) femur; (c) rectus femoris muscle; (d) vastus lateralis and intermedius muscle; (e) vastus medialis muscle

Full size image

9.4 Discussion and Conclusion

We presented an atlas-based 2D-3D intensity volume reconstruction approach, which to our knowledge is probably the first attempt to derive patient-specific musculoskeletal structures in the lower extremity. Our method has the advantage of combining the robustness of 2D-3D landmark reconstruction with the smoothness properties inherent to B-spline-based 3D regularization. In order to preserve knee joint structure, we proposed an articulated 2D-3D reconstruction strategy which can derive the anatomically correct reconstruction results, and we also investigated the reconstruction of three thigh muscles via the proposed reconstruction pipeline, which holds the potential to be used in the clinical routine in future. The comprehensive results from a set of experiments demonstrated the efficacy of this 2D-3D reconstruction method.