PURE: Panoramic Ultrasound Reconstruction by Seamless Stitching of Volumes

Abstract

For training sonographers in navigating, acquiring, and interpreting ultrasound images, virtual-reality based simulation offers a safe, flexible, and standardized environment. In data-based training simulations, images from a-priori acquired volumes are displayed to the trainee. To understand the relationship between organs, it is necessary to allow for free exploration of the entire anatomy, which is often not possible with the limited field-of-view (FOV) of a single ultrasound volume. Thus, large FOV ultrasound volumes are of paramount importance. Combining several volumes into one larger volume has also potential utility in many other applications, such as diagnostic and operative guidance. In this work, we propose a method for combining several ultrasound volumes with tracked positions into a single large volume by stitching them in a seamless fashion. For stitching, we determine an optimal cut interface such that each pixel value comes from a single image; preserving the inherent speckle texture and preventing any blurring and degradation from common mean/median binning approaches to combining volumes. The cut interface is found based on image content using graphical models optimized by graph-cut. We show that our method produces panoramic reconstructions with seamless transitions between individual 3D acquisitions. Regarding standard deviation in homogeneous regions we get 1–19% loss of ultrasound texture compared to small 3D volumes while mean value interpolation gives a loss of 15–68%. The histograms of our reconstruction match the original histograms of the small 3D volumes almost perfectly with a $\chi ^2$-distance of less than 0.01.

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Ultrasound is a safe and low-cost imaging modality. However, acquisition and interpretation of ultrasound images heavily rely on the experience and skill of the clinician. For training these skills, volunteers, cadavers, and phantoms all have associated ethical and realism issues. Virtual-reality based simulated training, on the other hand, offers a safe, flexible, and repeatable environment for the training of ultrasound imaging. Compared to ray-tracing based techniques, data-based ultrasound simulations provides relatively high image realism, where image slices are interpolated during simulation from a-priori acquired ultrasound volumes [1, 4, 8, 11], which can also accommodate interactive tissue deformations [7].

With 3D transducers, it is relatively easy to acquire volumetric ultrasound data. However, with the field-of-view (FOV) of a single 3D volume, it is not possible to freely explore within a large anatomical region (e.g., a third trimester fetus), which is an essential skill to be learned for diagnosing patients. The acquired simulation volume needs to ideally span, for example, the entire abdomen to provide a realistic scene for training. Large-FOV ultrasound volumes are also potentially beneficial for diagnostic applications. For instance, standardized, large-FOV ultrasound volumes of the breast can now be acquired using ABUS and ABVS platforms.

Gee et al. [6] proposed to use freehand 2D acquisition to reconstruct a 3D large-FOV volume. Despite the simplicity of acquisitions, the reconstructions, however, are often distorted and not up-to training standards due to the challenges in aligning several degrees of freedom for each frame without any out-of-plane information. In contrast, image volumes acquired using 3D transducers are inherently consistent. Accordingly, only continuity between such volumes needs to be ensured. This latter approach was followed in [3, 12, 14, 16]. These works mainly investigate registration strategies to fine-tune the alignment of volumes, while combining the images using simple binning (e.g., averaging) and interpolation methods. Herein we focus on such combination strategy itself, for which we propose the seamless stitching of ultrasound volumes based on graph-cuts.

For aligning the volumes, we use tracked transducer positions. We first apply a pressure compensation step to eliminate any tissue deformation resulting from probe pressure during acquisition. Then, one needs to determine which intensity values to assign to voxels in the overlapping parts of acquired volumes. Simple approaches are setting the voxel intensity to mean or median value of any overlapping voxels (binning) [15]. As one can imaging and we show later, such simple methods lead to the loss of typical ultrasound speckle texture due to blurring and emphasizes the overlap boundaries as artifacts in the resulting volumes. One alternative approach could be to divide the overlapping region with a plane, where on each side the ultrasound content is taken from the corresponding input volume. Such stitching plane, however, becomes visible as an artifact in the reconstructed volumes, since the speckle pattern is interrupted and will not fit from both sides. We hereby propose a non-planar stitching interface (surface) based on image information. To find the interface that will yield a seamless transition between the volumes, we devise a graphical model based on image content from overlapping parts of volumes. We solve this using graph-cuts, resulting in a 2D cut manifold that divides the overlap into two regions where image content can be used from corresponding input volumes.

2 Methods

2.1 Position Tracking

For data acquisition we use a mechanically-swept transducer equipped with a 6-DOF electromagnetic tracking sensor, in order to get an initial position and orientation of the volumes. Sensor-to-image calibration is done with an N-wire phantom using the method of [10]. The probe position $p_i$ is recorded at the time of each individual ultrasound volume acquisition i. Following each acquisition of probe position, another position $r_i$ is also recorded after having lifted the probe normal to body surface until it is barely touching the skin. This latter position is used in the pressure correction step. Neighboring volumes are collected to have overlapping image regions and to leave minimal to no gap between them.

2.2 Pressure Correction

For stitching, the volumes have to reside a common coordinate frame and deformation state. For good skin contact during acquisitions, pressure is applied on the transducer, deforming anatomy differently for each volume. We thus chose to bring the volumes to a nominal (undeformed) state, before stitching them, using the model-based pressure correction procedure of [5]. For each image volume i, first, a box-shaped tetrahedral mesh covering the imaging field-of-view is generated centered on top at the nominal position $r_i$. A 3D probe model is then translated from $r_i$ to $p_i$ to simulate the indentation by compressing the FEM nodes falling inside this model onto the probe surface using displacement constraints. The inverse of this deformation field is then applied on the acquired volume i to “undeform” it to a nominal uncompressed state. We incorporate the inverse transformation into the scan-conversion process to avoid interpolation artifacts. The depth of the meshed volume is assumed to be sufficiently large for stresses caused by the surface compression to be negligible near the bottom surface, accordingly setting the nodes at the far end of the imaging volume fixed in all axes. A homogeneous tissue content with linear isotropic material is used. Note that we here use empirically-set boundary conditions to keep the level of complexity low. With more sophisticated methods like [13] the results may even improve. After the pressure correction above, the anatomical structures should better align, especially near the surface. Finally one needs to define the orientation of a Cartesian coordinate frame for the final stitched volume from combining several individual pressure-corrected volumes. We choose the orientation of the most central volume assuming this volume contains image content of great interest. To pick the most central volume, we first compute the center-of-mass $c_i$ for all v volumes and then their centroid as $c=\sum _i\frac{c_i}{v}$. The global coordinate frame is then chosen as the coordinate frame of the volume closest to this centroid, i.e. $i=\arg \min _i|c_i-c|$. Before any stitching, all undeformed volumes are interpolated on a regular grid in this coordinate frame covering the bounding box of all available image data.

Fig. 1.

Illustration of a graphical model in 2D showing two volumes. The green and blue areas represent voxels with image values from one or the other volume. The overlapping volume after interpolating both volumes on a regular grid in the common coordinate frame, is marked with green and blue stripes. The boundary voxels, connected to the source or the sink, are shown as thick lines. (Color figure online)

2.3 Image Stitching

Consider the overlapping volume that include image voxels with values from both of the two input volumes to be stitched as shown in Fig. 1. For the optimal transition interface (surface) for a seamless transition, we define a graphical model within the overlapping volume as a graph G(N, E) where each node $n\in N$ represents an overlapping voxel connected by edges $e\in E$ to its six neighboring voxels in 3D. Each edge connecting two neighboring voxels $\mathbf {x}$ and $\mathbf {y}$ in overlapping volumes $V_1$ and $V_2$ is assigned a capacity (potential) based on a transition quality metric similarly to [9]. This edge potential P is based on neighbours image intensity as well as the image gradient between them as

$$\begin{aligned} P(\mathbf {x},\mathbf {y})=\frac{\Vert V_1(\mathbf {x})-V_2(\mathbf {x})\Vert +\Vert V_1(\mathbf {y})-V_2(\mathbf {y})\Vert }{\Vert \nabla _{V_1}^e(\mathbf {x})\Vert +\Vert \nabla _{V_1}^e(\mathbf {y})\Vert +\Vert \nabla _{V_2}^e(\mathbf {x})\Vert +\Vert \nabla _{V_2}^e(\mathbf {y})\Vert +\varepsilon }, \end{aligned}$$

(1)

where $\nabla _{V_i}^e(.)$ is the gradient in image volume $V_i$ along the graph edge e. To avoid division by zero, we add a small value $\varepsilon = 10^{-5}$ to the denominator. If the overlapping images match intensity-wise at both nodes x and y, then this is an ideal place for stitching and it is permitted by the vanishing nominator. In case images do not match, cuts along image edges are encouraged by the denominator. Since intensity changes are already anticipated at natural edges, even if mean intensities on either side do not match, such seams are not likely to be visible. A graph is constructed only for the overlapping voxels, with source s or sink t of the graph connected to all boundary voxels marked as thick blue and green lines in Fig. 1. Note that since this is an undirected graph, i.e. $P(\mathbf {x},\mathbf {y})=P(\mathbf {y},\mathbf {x})$ for all neighboring voxels $\mathbf {x}$ and $\mathbf {y}$, the transition interface (solution) is independent of the order of input volumes (source and sink). Finally, the minimum cost cut of this graph is found using [2], giving a partition of G such that $\min \sum _{\mathbf {x}\in V_1,\mathbf {y}\in V_2|e=(\mathbf {x},\mathbf {y})\in E}P(\mathbf {x},\mathbf {y})$ with $s\in V_1$ and $t\in V_2$.

The solution labels for each voxel indicate whether its intensity is to be assigned from volume $V_1$ or volume $V_2$. We noticed that even with optimal cuts, there can still be artifacts along stitched interfaces where no suitable seams exists, e.g. due to a quite small overlap and view-dependent artifacts like shadows. We reduce these artifacts by blending the volumes across the seam using a sigmoid function with a small kernel. For stitching more than two volumes we start with the volume closest to the center-of-mass of all volumes and then iteratively merge in the volumes in order of descending proximity.

3 Results and Discussion

We first demonstrate a typical cut resulting from our method, Panoramic Ultrasound REconstruction (PURE), on a 2D sample in-vivo data in Fig. 2. This shows two B-mode images of the uterus, once prior to speculum placement and once afterward. These significantly deformed states present a challenging scenario to showcase our algorithm. This task also demonstrates a potential use of stitching for the seamless modification of ultrasound B-mode content, such as for artificially introducing tools, anatomical alterations, and even pathology. Images are considered spatially fully overlapping, where the blue (left) mask is taken from the image without speculum and the green right mask from the image with speculum. The middle yellow part is then reconstructed, using its borders with blue and green areas as the sink and the source. In comparison to results using mean value reconstruction and a simple transition exactly along the center line, PURE is seen in Fig. 2(f) to present a realistic image free from reconstruction artifacts, despite the relatively poor overlap of anatomy. In order to present the almost-invisible PURE stitching interface, we also depict the seam in Fig. 2(g).

Fig. 2.

A 2D stitching example showing the uterus (a) without and (c) with a speculum, using a PURE mask (b). The bottom row shows 3 images with the blue areas from (a) and the green areas from (c) and the middle yellow area been reconstructed using (d) mean value, (e) merging “bluntly” at a cutting line in the middle, and (f) using PURE. The cutting curve from graph-cut is depicted in (g). (Color figure online)

For evaluation we applied our reconstruction technique on both the CIRS fetal ultrasound biometrics phantom (model 068) seen in Fig. 3 and the CIRS female pelvic ultrasound training phantom (model 404A) using an Ultrasonix SonixTouch machine with a convex 4D probe (4DC7-3/40). For both scans we recorded the positions tracked electromagnetically using an Ascension trakSTAR system.

Fig. 3.

Fetal ultrasound phantom, 3D ultrasound volumes showing only parts of the phantom, and 3D view of PURE covering the entire fetus.

For comparing reconstructions, Figs. 4(a) and (b) present PURE as well as alternative methods using mean and median value binning. With the latter methods, smoothing and deterioration of typical ultrasound speckle texture is apparent, especially in the blown-up insets of the fetus reconstructions. In addition to preserving speckle texture, view-dependent artifacts like shadows and reflections are also lost in the mean and median reconstructions, as marked by the arrows in the sagittal pelvic images. PURE is seen to accurately preserve these fundamental ultrasound artifacts, essential also for the training of image interpretation. For the pelvic phantom the boundaries of the small volumes are very obvious in mean and median result as pointed to by arrows in the top left image of Fig. 4(b). In the PURE result the single volumes are not apparent at all which means a really seamless transition.

Fig. 4.

Comparison of large FOV reconstructions. Both phantom measurements has 6 volumes as input. The insets in the fetus phantom show the fine speckle pattern preserved by our method PURE. (a) fetal phantom. (b) pelvic phantom. (Color figure online)

To demonstrate the need for pressure correction, the right-most column of Figs. 4(a) and (b) show the stitching results without pressure correction. Elasticity parameters for pressure correction were set to a Young’s modulus of 10 KPa and a Poisson’s ratio of 0.45 in line with the phantom material composition. It is clearly visible that the surface of the non-corrected case is deteriorated by several probe footprints as pointed out by arrows in the top right image of Fig. 4(b). This hinders the alignment of the skin of a virtual or a physical model when integrating the stitched volume into a image-based ultrasound simulator. Note that the top part of the sagittal slice of the pelvic phantom shows a failure case marked by a red circle. Since bladder is a highly deformable organ, the corresponding volume is severely distorted and the pressure correction could not align the volumes sufficiently.

The phantoms contain regions with homogeneous echogenecity, which we used to assess the speckle appearance of the reconstructed volumes. We manually selected largest-possible 3D bounding boxes within homogeneous regions of the reconstructed phantom volumes, and applied intensity statistics to compare reconstructed volumes to original acquired images. This led to 7.6 K, 14.5 K and 41.3 K voxels, respectively, inside the abdomen of the fetus model, and inside and outside the uterus of the pelvic model. For each bounding box we characterize the texture for the original volumes $V_i$ by computing the standard deviation $\sigma _i$. We compared the standard deviation $\sigma$ within the bounding box of the reconstructed volumes to the groundtruth standard deviation, set as the mean value of the standard deviations in the small volumes $\sigma _{GT}=\sum _i\frac{\sigma _i}{v}$. Normalizing $\sigma$ to the ground-truth baseline, it is seen that the $\sigma$-error of the median approach is 2.6 to 12 times poorer than that of PURE (which is merely 1 % difference to our ground truth in fetus phantom). The large errors for the outside uterus part of pelvis phantom can be attributed to the more significant directional shadowing and enhancement artifacts in those regions. Additionally, the histograms of these homogeneous regions are compared by computing the $\chi ^2$ distance. The histograms of PURE match the histogram of the original images up to 100 % ($\chi ^2$ distance equal to zero) if all data is taken from one volume.

Table 1. Comparison of changes in ultrasound texture by combining several volumes

For few image parts with poor overlap there may exist no obvious seamless interface, e.g. the cut pointed by the arrow in Fig. 5. Nevertheless, the proposed sigmoidal blending (with a kernel of 3 voxels) is seen to successfully remove such “sharp” cuts locally. Note that, in contrast to mean-binning everywhere, such a small sigmoid kernel acting locally affects only a tiny part of the entire volume, preserving the overall quality of the volume. As can be seen from the results in Figs. 2, 4(a) and (b), the effect of such local blending is indiscernible.

Fig. 5.

Seamless stitching is improved by postprocessing by sigmoidal blending.

Finally we acquired in-vivo data of a volunteer’s abdomen in the gynecological setting using a GE Voluson E8 machine with a convex 4D probe (RAB4-8-D). Positions were tracked electromagnetically using an Ascension trakSTAR system as for the phantom data. Figure 6 shows the promising results of stitching six volumes into a large FOV reconstruction. Organs and structures match very well and the seams are invisible compared to mean or median value interpolation while the texture is preserved. As for the pelvic phantom, the boundaries of the small volumes are again visible in mean and median results whereas they are not distinguishable in the PURE result.

Fig. 6.

Comparison of volumetric large FOV reconstructions of in-vivo data. The measurement has 6 volumes as input.

Stitching two volumes, each of size $240\times 210\times 250$, needs 4 min with current implementation on an Intel i7-4770 K processor, where roughly 98 % of computation time is spent on the min-cut solution. In comparison, combining the volumes by mean-binning needs 4 s.

4 Conclusions

We have introduced a method for generating panoramic ultrasound volumes beneficial for training simulation and diagnostic purposes. Our method, called PURE, is based on stitching 3D volumes at optimal cut interfaces determined by graph-cut from image content. In contrast to conventional mean and median value reconstruction, PURE prevents the deterioration of typical ultrasound speckle texture in the overlapping regions. PURE necessitates more computation time compared to algebraic approaches such as median and mean, which is acceptable for simulation purposes where the large-FOV reconstruction is required offline only as a preprocessing step. A further application of our algorithm in simulation framework is to edit image content as demonstrated by the 2D stitching example in Fig. 2, where image parts are shown to be replaced by content from another image. This can simplify the generation of several pathological cases, which is a major bottle-neck of data-based ultrasound simulation. Similarly to algebraic reconstruction, PURE results may also suffer from stitching artifacts when organs are not aligned between given images, e.g. due to tracking errors and deformation. An image registration stage (e.g. [3, 12, 14, 16]) prior to stitching will be investigated next for improving reconstruction results. This should improve the seamless transition especially in case of inaccurate tracking data. Note that a major motivation for stitching is simulated training, which does not require a fully unsupervised image reconstruction, but a clinician can easily check the quality of reconstructions. For diagnostic application the transition surface along with a confidence value, e.g. cutting cost from graph-cut, can be displayed to the clinician to prevent misleading diagnosis.