An Optical Flow Approach to Assessment of Ventricular Shape Change Based on Echocardiography

Abstract

The quantitative analysis of cardiac motion from echocardiographic images helps clinicians in the diagnosis and therapy of patients suffering from heart disease. Quantitative analysis is usually based on TDI (Tissue Doppler Imaging) or speckle tracking. These methods are based on two techniques which to a large degree are independent—the Doppler phenomenon and image sequence processing, respectively. Herein, to increase the accuracy of the speckle tracking technique and to cope with the angle dependency of TDI, a combined approach dubbed TDIOF (Tissue Doppler Imaging Optical Flow) is proposed. TDIOF is formulated based on the combination of B-mode and Doppler energy terms minimized using algebraic equations and is validated on simulated images, a physical heart phantom, and in-vivo data. It was observed that the additional Doppler term is able to increase the accuracy of speckle tracking, compared to two popular motion estimation and speckle tracking techniques (Horn-Schunck and block matching methods). This observation was more pronounced when noise was present. . The magnitude and angular error for TDIOF applied to simulated images when comparing estimated motion with ground-truth motion were 15 % and 9.2 degrees/frame, respectively. The magnitude and angular error for images acquired from physical phantoms were 22 % and 15.2 degrees/frame, respectively. As an additional validation, echocardiography-derived strains were compared to tagged MRI-derived myocardial strains in the same subjects. The correlation coefficient (r) between the TDIOF-derived radial strains and tagged MRI-derived radial strains value were 0.83 ($\mathrm{P}<0.001$ ). The correlation coefficient (r) for the TDIOF-derived circumferential strains compared to the tagged MRI-derived circumferential strains were 0.86 ($\mathrm{P}<0.001$ ). The comparison of TDIOF-derived and block matching speckle tracking and Horn-Schunck optical flow strain values using student t-test demonstrated superiority of TDIOF (95 % confidence interval, $\mathrm{P}<0.001$ ).

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Cardiovascular Disease (CVD) is the leading cause of death in the modern world. The mortality rate associated with CVD was estimated to be 17 million in 2005 and continues to be ranked as the top killer worldwide. CVD is the result of under-supply of the cardiac tissue and can lead to malfunction of the involved myocardial territories and manifest as hypokinesia or akinesia. Several imaging methods such as X-ray CT, MRI, and Ultrasound have been used for visualization of the heart function. MRI and X-ray CT provide excellent spatial resolution but the cost and lack of wide-spread availability cause challenges in the clinical settings. Echocardiography is a popular technique for cardiac imaging due to its availability, ease of use, and low cost. Echocardiography shows the motion and anatomy of the heart in real time, enabling physicians to detect different pathologies. However, analysis of motion of the myocardium in echocardiographic images is based on visual grading by an observer and suffers from inter and intra-observer variability. To overcome the inter- and intra-observer variability, computerized image analyses can help by quantitatively interpreting the data. To that end, cardiac image processing techniques, mainly categorized as segmentation and registration, have been widely used for assessing the regional function of the heart [1–3]. To perform such analysis, two independent techniques, have been utilized; these are TDI (Tissue Doppler Imaging) and speckle tracking. TDI computes the tissue motion based on the Doppler phenomenon and is dependent on the angle of insonification. Speckle tracking, on the other hand,is an image processing method based on the analysis of the ultrasound B-mode or RF images. B-mode based algorithms are robust to the variation of the transducer angle but rely entirely on the properties of echocardiographic images which may be noisy or inaccurate. The physical principle underlying B-mode and TDI are to a large degree independent and therefore for myocardial motion estimation carry complementary information [4, 5].

Table 1 Description of some of the current methods used in motion detection in echocardiography images

Many methods such as optical flow [6], feature tracking [7], level sets [8], block matching [9], and elastic registration [10] have been utilized for quantitative assessment of myocardial motion in B-mode images. Table 1 shows a description of some of the current methods used in motion estimation in echocardiographic images [6–21]. Suhling et al. [6] integrated rigid registration in an optical flow framework in order to detect myocardial motion from 2D echocardiographic images. B-spline moments invariants were applied to echo images to achieve invariance to the translation and rotation. The motion estimation algorithm was then applied to the B-spline moments of the image instead of the image intensity in a coarse to fine strategy and was validated using open chested dogs after ligation of a coronary artery. Additional validations were performed on simulation and phantom images. Ellen et al. [10] used elastic registration on 3D B-mode echocardiography images to extract myocardial motion and strain values. The method was validated using simulated and real ultrasoundimages. Esther-Leung et al. [11] proposed two different methods (1. model-driven, 2. edge-driven) for tracking the left-ventricular wall in echocardiographic images. Their approach was motivated by the fact that in echocardiography images, visibility of the myocardium depends on the imaging view; so the myocardium may be, partly, invisible to the beam. Their technique relied on a local data-driven tracker using optical flow applied to the visible parts of the myocardium and a global statistical model applied to the invisible parts. It was concluded that the shape model could render good results for both the visible and the invisible tissues in ultrasound images. Myronenco et al. [12] proposed the so-called Coherent Point Drift (CPD) technique for myocardial motion estimation, constraining the motion of the point set in the temporal direction for both rigid and nonrigid point set registration. A set of point distribution was computed based on endocardium and epicardium locations. The point set was modeled with a Gaussian mixture model (GMM). The GMM centroids were updated coherently in a global pattern using maximum likelihood to preserve the topological structure of the point sets. A motion coherence constraint was added based on regularization of the displacement fields. The purpose of regularization was to increase the motion smoothness.

Most of the motion estimation techniques developed thus far, are either based on TDI or B-mode. Recently, Garcia et al. [22] considered the combination of cardiac B-mode images and intra-cardiac blood flow data for computing the blood flow motion in the heart using continuity equation and mass conservation in polar coordinates. Their paper focused on the blood flow computation and did not consider the cardiac tissue displacements. Dalen et al. [23] and Amundsen et al. [24] previously combined TDI with speckle tracking by integrating TDI in the beam direction and speckle tracking in the direction lateral to the beam. However, this method discarded the speckle tracking data in the beam direction. The authors reported that they were unable to improve the motion estimation performance compared to speckle tracking techniques. In this paper, we propose integration of tissue Doppler and speckle tracking within a novel optical flow framework, we call TDIOF (Tissue Doppler Optical Flow). Our experimental results indicate that TDIOF outperforms both TDI and speckle tracking approaches.

The organization of the rest of the paper is as follows: in Sect. 2, we review the mathematical and algorithmic basis for the proposed method. Section 3 is a description of datasets used for validation of the proposed method. These include computational simulations, US data collected in a cardiac phantom, and in vivo data collected in patients recruited from the echocardiography laboratory at the Robley Rex VA Medical Center to our IRB-approved study. In Sect. 4, strain computations are discussed and, in Sect. 5, results from validation of the proposed method are described. Finally, in Sect. 6 we discuss observations related to TDIOF and our findings.

2 Methods and Materials

TDI and B-mode speckle tracking are different in both their physical underpinning and data type. In speckle tracking, tissue motion is determined from motion of speckles in Ultrasound images–typically, using a block matching approach applied to 2-D B-mode images [give references]. Although speckle tracking provides both components of the motion in the spatial domain, it is based on noisy B-mode images. TDI on the other hand only computes the velocity of tissue in one direction as moving towards (displayed as red) or away (displayed as blue) from the transducer. This means that the computed motion is the projection of the real motion in the direction of the transducer and therefore TDI is angle dependent. In this section, we first describe a novel energy minimization framework for estimation of myocardial motion from B-mode images which incorporates a velocity constraint from TDI.

The proposed method is based on optimization of three energy functions: (1) intensity constancy assumption, (2) velocity smoothness, and (3) similarity with Doppler data. The framework is minimized using an incremental algebra in method [25, 26] as described in Appendix A. In order to show the performance of TDIOF, it is compared to two popular motion detection techniques Horn-Schunck optical flow [27] and block matching [28] (Appendix B). Block matching is utilized in several commercial software platforms [29].

3 Validations

3.1 Simulated Computerized Phantom

Echocardiographic images are the result of the mechanical interaction between the ultrasound field and the contractile heart tissue. Previously, we reported on development and use of an ultrasound cardiac motion simulator [29]. In the current study, we utilized the COLE convolution based simulation technique reported in [30]. The significance of an Ultrasound cardiac motion simulator is the availability of both echocardiographic images as well as the actual ground-truth vector field of deformations.

Table 2 The 13 k-parameters of the Art’s kinematic model for left-ventricular deformation used in our cardiac US motion simulator

A moving 3D heart was modeled based on a pair of prolate-spheroidal representations and used for the ultrasound simulation. The 3D forward model of cardiac motion was simulated using 13 time-dependent kinematic parameters of Arts et al. [30] (see Table 2). The evolution of the 13 kinematic parameters was previously derived by Arts following a temporal fit to actual location of tantalum markers in a canine heart [31]. In Arts’ model, seven time-dependent parameters are applied to define the ventricular shape change, torsion, and shear while six parameters are used to model the rigid-body motions. To simulate the Ultrasound imaging process, scatterers were randomly distributed in the simulated LV wall and the motion prescribed by Arts’ model was used to move the ultrasound scatterers. To determine Ultrasonic B-mode intensities, the COLE method was used [30]. COLE is an efficient convolution-based method in the spatial domain, producing US simulations by convolving the segmental PSF (point spread function) with the projected amplitudes of the scatterers [29] with the segmental PSF derived using Field II [30, 32]. In order to model the Doppler Effect, the frequency of the RF signal was shifted in the frequency domain based on the attributed ground truth motion vector and mixed with additive Gaussian noise. If the velocity of the particle is $v$, ultrasound velocity is $c$, and transducer frequency is $f$, then the frequency shift is:

$$\begin{aligned} \Delta f=\frac{2vf}{c} \end{aligned}$$

(1)

The resolution of the first simulated sequence was 0.1 mm/pixel for both B-mode and TDI images and included 14 mid-ventricular temporal frames in the axial orientation. In order to analyze the robustness of the method to noise, another set of simulated images were produced by adding Gaussian noise of 1.12 db to the noise-less data

3.2 Physical Cardiac Phantom

As described in [30], a physical cardiac phantom was built in-house, suitable for validation of echocardiographic motion estimation algorithms. Here, a brief description of this phantom is provided. To manufacture the phantom, a cardiac computerized model was used to build an acrylic based cardiac mold. A 10 % solution of Poly Vinyl alcohol (PVA) and 1 % enamel paint were used as the basic material. PVA has the ability to mimic cardiac elasticity, ultrasound and magnetic properties. The solution was heated up to 90 $^{\circ }$C. Consequently, it was poured into the cardiac mold and gradually exposed to the temperature of $-20$ $^{\circ }$C until it froze. The mold and the solution were kept in that temperature for 24 h. Finally, the mold and the frozen gel were gradually exposed to the room temperature. At this point, the normal heart phantom has passed one freeze-thaw cycle.

An additional model consisting of the left and right ventricles but with a segmental thin wall in the LV was used to build an additional mold for a pathologically scarred heart. The thinner wall was designed to mimic an aneurysmal, dyskinetic wall. Three PVA-based inclusions were separately made as a circle; slab and cube using nine, six and three freeze-thaw cycles respectively. Each freeze-thaw cycle decreases the elasticity of the heart mimicking scarred myocardium. The attenuation of the PVA and speed of sound increase after each freeze-thaw cycle.The cylindrical, slab like and cube like objects were placed in the mold in different American Heart Association cardiac segments [33]. Subsequently, the PVA solution was added to fill the rest of the space in the mold. After one freeze-thaw cycle, the abnormal heart consisted of a background of normal texture with one freeze-thaw cycle plus three infarct-mimicking inclusions having 10, 7 and 4 freeze-thaw cycles. The speed of sound in PVA is 1527, 1540, 1545, and 1550 m/s and ultrasound attenuation is 0.4, 0.52, 0.57, and 0.59 db/cm for 1, 4, 7 and 10 freeze-thaw cycles. The parameters of the synthetic phantom was adjusted based on the previous phantom studies [34].

A mediastinal phantom that provides the ability to acquire trans-esophageal images was manufactured using another mold. A solution of 50 % water and 50 % glycerol was used to mimic the blood. Finally, a syringe was used to manually force the fluid inside and outside the phantom for contraction and expansion. The enamel paint particles are robust scatterers and can generate reliable markers on the B-mode image. Since each marker is not restricted to just one pixel, the center of the mass of each manually segmented marker is considered as landmark. The displacements of the landmarks are compared to the computed motion field for the validation purposes. Figure 1 shows the cardiac phantom and the acquired phantom images using ultrasound and MRI.

Fig. 1

Seventeen AHA prescribed segments for the heart. a Basal SAX view, b mid-LV SAX view, c apical SAX view (AS antero-septal, A anterior, L lateral, P posterior, I inferior, IS infero-septal) [35, 37, 38]

3.3 Patient Studies

Two separate sets of data were utilized for in vivo validations (sets A and B). Set A contained 15 patients and was used for manual tracking validation. Set B was a joint echo and tagged MRI set and was used for both manual tracking and comparison with tagged MRI (as will be discussed in Sect. 3.3.2) (Fig. 2).

Fig. 2

a A picture of the two-chamber model. b TDI image of the moving phantom during balloon inflation. c A static slice of the phantom using T1 weighted FFE. The arrow points to the aneurysm (the thin ventricular wall). d A static slice of the phantom using balanced FFE (1 LV, 2 RV, 3 cylindrical inclusion, 4 slab-like inclusion, 5 cube like inclusion, 6 mediastinum and mediastinal structures)

3.3.1 Set A: Echocardiography Studies

Data from fifteen subjects who had already undergone echocardiographic imaging as part of their diagnostic evaluation were deidentified and transferred to the laboratory following IRB approval. The data included 13 male, 4 female, average age $52.9 \pm 7.3$, consisting of hypertension (8 cases), Coronary Artery Disease (4 cases), Left Ventricular Hypertrophy (4 cases), Congestive Heart Failure (1 case), Chronic Obstructive Pulmonary Disease (2 cases), Diabetes Mellitus (2 cases) and smokers (1 case). 2D echocardiography (short-axis, long-axis, four-chamber, two-chamber B-mode with TDI. At the University of Louisville Hospital’s echocardiography laboratory, Echocardiographic images are acquired with a commercially available system (iE33, Philips Health Care, Best, The Netherlands) using a S5-1 transducer (3 MHz frequency) and the operator is free to change the gain and filter as needed. The full data set included two-chamber, three-chamber, four-chamber, and long-axis views.

3.3.2 Set B: Echocardiography-MRI Studies

The prospective protocol for patient selection and imaging was approved by the Institutional Review Board of the Robley Rex Veterans’ Affairs Medical Center, and a written informed consent was obtained from patients. Eight male subjects were prospectively recruited to the study with average age $54.6 \pm 8.5.$ The subjects included hypertension (5 cases), Coronary Artery Disease (2 cases), Congestive Heart Failure (1 case), Chronic Obstructive Pulmonary Disease (1 case), Diabetes Mellitus (3 cases) and smokers (4 cases).The imaging protocol included a primary 2D echocardiography including short-axis, long-axis, three-chamber, four-chamber and two-chamber B-mode and TDI imaging as well as simultaneous B-mode/TDI imaging (two-chamber, three-chamber, four-chamber, long-axis). At the Robley Rex Veterans Affair Medical Center’s echocardiography laboratory, Echocardiographic images are acquired with an iE33 commercial echocardiography system (Philips Health Care, Best, The Netherlands) using a S5-1 transducer (3 MHz frequency) and the operator is free to change the gain and filter as needed.

Following Ultrasound imaging, cine and tagged MRI data were collected in all subjects. Tagged MRI data acquisition was performed using Philips Achieva, TFE/GR sequence, TE/TR 2/4 ms, Flip Angle 15, spatial resolution $1.25 \times 1.25$ mm, slice thickness 8 mm, and spatial size $256 \times 256 \times 8$ pixels. In all subjects, both echocardiography and MR imaging were performed within two hours to decrease any confounding events that could cause discrepancy between wall motion studies in echocardiography and MRI. MRI was performed immediately after the echocardiography. In order to ensure that the B-mode and TDI images were matched, B-mode and TDI images were simultaneously acquired. Additionally, subjects were asked to hold their breath during data collection.

3.4 In-Vivo Comparison of TDIOF-Derived Strains with Strains from Tagged MRI

Tagged MRI [35] is known to provide highly accurate displacement fields in the systolic portion of the cardiac cycle while the tags last. We analyzed the strain field in echocardiography and tagged MR images of slices similar in location in the two modalities in set B. In selecting corresponding slices, qualitative anatomical landmarks such as the papillary muscles and cardiac contours as well as cine MRI images were utilized. Anatomical landmarks such as endocardial shape and papillary muscle were used to locate the appropriate short axis sections of the heart. The recently proposed SinMod technique [36] was then used to derive displacements from tagged MRI data in the first few systolic phases of the cardiac cycle, while the tags persisted. SinMod is an automated motion estimation technique for tagged MRI that models the pixels as a moving sine wavefront.Since no pixel to pixel mapping between echo and MR images was known, the ventricular geometry from 2-D echo and tagged MRI was divided into 17 segments following the American Heart Association’s recommendations. Subsequently the averaged Lagrangian strain for each of the 17 heart segments were compared between the two modalities. Since the frame rate of echo and MRI is not the same and the heart rate may change, it was necessary to align the images in the temporal dimension. This was done by spline interpolation of the measured strain data in the time domain.

4 Strain Analysis

Strain is a measure of deformation of the cardiac tissue. With $I$ representing the identity matrix, the Lagrangian strain tensor at a given myocardial point and for a specific time point can be expressed as:

$$\begin{aligned} E=\frac{1}{2}(F^{T} F - I) \end{aligned}$$

(2)

where the elements of the deformation gradient tensor, F, are:

$$\begin{aligned} F = \begin{pmatrix} \frac{\partial x}{\partial X} &{} \frac{\partial x}{\partial Y} &{} \frac{\partial x}{\partial Z} \\ \frac{\partial y}{\partial X} &{} \frac{\partial y}{\partial Y} &{} \frac{\partial y}{\partial Z} \\ \frac{\partial z}{\partial X} &{} \frac{\partial z}{\partial Y} &{} \frac{\partial z}{\partial Z} \end{pmatrix} \end{aligned}$$

(3)

while $x=X+V(X), X$ represents the spatial coordinates in the undeformed coordinates (typically taken to be the end-diastolic frame), and $V(X)$is the accumulated motion vector at the corresponding spatial location relative to the undeformed state . For the echocardiography data, the reference frame for the strain computation was considered to be the end diastolic frame and was selected based on ECG trigger. The deformation field was then computed between each two frames and was added to the motion field from the previous frame in order to measure the accumulated deformation and strain . Since the deformation field of the consecutive frames do not represent the motion of the same pixels, spline interpolation was used to align the deformation fields. For the tagged MRI data, the end-diastolic frame was always the first acquired image which is collected immediately after the R-wave trigger.

The normal strain in the direction of the unit vector n can be calculated from the Lagrangian strain tensor through the quadratic form ${n}^{T}n$, where n is a unit vector and can point to any direction on the unit sphere. Due to the geometry of the left ventricle, the normal strains are usually calculated in radial, circumferential, and longitudinal directions.

Regional analysis is performed on 17 American Heart Association (AHA) prescribed segments. Figure 7 shows the different segments. The acronyms stand for antero-septal (AS), anterior (A), lateral (L), posterior (P), inferior (I), and infero-septal (IS). For a review of topics related to determination of strain from cardiac images, the reader is referred to [35, 37, 38].

5 Results

As noted in Sect. 3, TDIOF was applied to three different datasets: simulated images, data collected in a physical phantom, and in vivo data (both set A and set B). To further elucidate the effect of the Doppler term, results from TDIOF were compared to Horn-Schunck (HS) optical flow (beta = 0 in Eq. (A.11)) and block-matching (BM) (see Appendix) with the latter being the basis for most commercial speckle tracking methods [13]. Since the performance of each technique depends on the parameters of the method, it was necessary to optimize the parameters. Based on simulated images, an exhaustive search was performed over the parameters of TDIOF, HS optical flow, and BM speckle tracking method (a large range was considered for each parameter) and the best values were selected experimentally. To analyze the performance of the techniques with different parameter settings, simulated images were compared to the next simulated frame after being warped using the estimated motion field. Relative mean absolute error was used for the comparison. Relative mean absolute error was computed as $\frac{1}{N}\sum _{i,j} {\left\| {\hat{{I}}-I} \right\| /I}$; where $I$ and ${\hat{I}}$ are the first and subsequent warped images, and $N$ is the total number of points. Figure 3 shows the performance of the TDIOF technique using different parameters. The methods were then applied to all the datasets using the resulting parameters: number of scales for multiscale implementation: $5, \alpha$ (smoothness weight): $2000, \beta$ (TDI similarity weight): 0.001, and $\sigma$ (penalizer parameter): 0.1. The parameters for the HS technique were set as follows: number of scales 5, and smoothness weight 2000.

Fig. 3

Performance of TDIOF using different parameters based on relative mean absolute error: X-axis is shown with a logarithmic scale in order to report a wide range of parameter settings. The performance of TDIOF is plotted versus smoothness coefficient for different TDI similarity coefficients ($\beta$). Changes of performance is evident when smoothness parameter ($\alpha$) changes. As seen from the plots, performance was more dependent on the smoothness and insensitive to the scale for the TDIOF term

5.1 Validations on Simulated Images

The simulated 3D cardiac model built based on Arts’ et al. [31] is shown in Fig. 4a. The deformation shown in the figure is that of a systolic motion. The 3D B-mode image deformation was computed based on [29] and was shown in Fig. 4b. Figure 4c shows the computed TDI using the simulated sequence—the red colors represent motion towards the transducer and the blue colors represents motion away from the transducer. Figure 5 shows application of TDIOF to simulated data and comparison with ground truth. Angular and magnitude error metrics were used for validation of the proposed technique as stated in Eqs. (4) and (5):

$$\begin{aligned} M\mathrm{agnitude \ Error} = \frac{1}{N} \sum _{{i},{j}} \Bigg |\frac{\Vert {\hat{V}}\Vert -\Vert V\Vert }{\Vert V\Vert }\Bigg | \end{aligned}$$

(4)

$$\begin{aligned} A\mathrm{ngular \ Error} = \frac{1}{N} \sum _{{i},{j}} \Bigg |Arc \cos \frac{\Big \langle V, {\hat{V}} \Big \rangle }{\Vert V\Vert \cdot \Vert V\Vert }\Bigg | \end{aligned}$$

(5)

Fig. 4

a Simulated cardiac model in diastole and systole. b A 3D simulated B-mode image based on COLE. c The computed tissue Doppler image using the simulated sequence

Fig. 5

a Results of TDIOF from a mid ventricular section of the 3D simulated echo data. b Ground truth motion field

where$V$ and $\hat{{V}}$ are the true and estimated displacement vectors and $N$ is the total number of vectors.

Table 3 TDIOF versus HS optical flow and BM speckle tracking when applied to simulated images

To quantitatively analyze the proposed method, averaged performance of TDIOF, Horn-Schunck optical flow, and block matching speckle tracking are reported in Table 3. The methods were applied to 14 simulated cardiac frames (one full cardiac cycle) of size $300 \times 300 \times 150$ pixels with and without noise. The error represents the angular or magnitude error averaged over all 100 slices and over all 14 temporal frames (averaged in both space and time). Please note that TDIOF was applied to 100 short-axis cross sections of the simulated heart. Table 3 illustrates that TDIOF has markedly improved performance on noisy images. Figure 6 shows the magnitude and angular error over one cardiac cycle for the 3 techniques—note that for each time point, the errors have been averaged over all spatial positions and all slices.It is evident that for all methods the errors are more pronounced in systolic frames compared to diastolic frames. This, we believe, is due to larger out of plane displacements causing errors for the 2-D method. From the figure, it can also be observed that, TDIOF outperforms Horn-Schunck optical flow and BM speckle tracking more significantly on noisy images.

Fig. 6

Comparison of (a) magnitude (Eq. (4)) and (b) angular error (Eq. (5)) over one cardiac cycle for different techniques

5.2 Validation on Data Collected in a Physical Phantom

In order to validate TDIOF on phantom data, the enamel markers on the B-mode images were manually segmented and the centers of mass of the markers were considered as landmarks. The error was computed on 128 landmarks over one cardiac cycle with 54 2D echocardiographic frames. As with simulated images, angular and magnitude errors were used to analyze the performance. The averaged magnitude and angular error of the landmarks for TDIOF, HS optical flow, and BMspeckle tracking are shown in Table 4. Figure 7 shows application of TDIOF and TDI to the physical phantom.

Table 4 Comparison of performance of TDIOF with HS optical flow and BM speckle tracking on physical phantom data

5.3 Validation on In Vivo Images

The algorithm was also evaluated in a similar way using in vivo images with 519 landmarks selected by an expert over 106 sets acquired from 23 patients. Landmarks were prominent regions in in vivo images such as speckles that could easily be detected. Each landmark was delineated and the center of mass of the landmark was defined to be the actual location. The average error for each of the three methods (TDIOF, Horn-Schunck, block matching) applied to in vivo data was classified per segment and is reported in Table 5. Figure 8a shows the application of the TDIOF technique to one four-chamber in-vivo B-mode study in systole. Figure 8b shows the end-systolic longitudinal strain map for the same patient.

Fig. 7

a TDIOF applied to phantom data in “systole”, b TDI forthe same phase

5.4 Preliminary Comparison of Strains from TDIOF and Tagged MRI

In this part of the study, radial and circumferential strains derived from TDIOF, HS optical flow, and BM speckle tracking were computed from B-mode echoand were compared to tagged MRI strains. Anatomical landmarks such as endocardial shape and papillary muscle locations were used to locate the corresponding short axis sections of the heart in tagged MRI and echocardiography. In addition, since the papillary muscles could not be easily visualized in the tagged studies, non-tagged cine MR images were used to better define the papillary muscles locations. Despite these efforts to ensure correspondence of the data, as alignment of the data based on landmarks could only be approximate (due to differences in slice thickness and identical view orientation in echo and MRI), and the results reported here should only be qualitatively interpreted.

Table 5 Comparison of displacement errors for TDIOF, HS optical flow, and BM speckle tracking on 17 AHA segments averaged over 23 patient data sets

Fig. 8

Application of TDIOF to four-chamber B-mode data during diastole. As expected, TDIOF-derived displacements are larger for the basal segments when compared to the apical segments

The image-derived strain values related to the same cardiac phase and the same sections of the same patient were compared by averaging the corresponding radial and circumferential strain values for each of the 17 AHA segments. For alignment, the short-axis tagged MR images were visually matched to the corresponding short axis echocardiographic images acquired from basal, mid-ventricular, and apical slices. Since the tag lines fade after systole, only the first 3–4 systolic tagged frames and the corresponding temporal extent in echo was considered in this analysis. Furthermore, since the number of the frames in echocardiography is several times that of tagged MRI data (roughly 20 tagged MR frames versus 4 echocardiographic frames during the cardiac cycle), the strain fields in echo images were interpolated using spline interpolation to match the systolic tagged MRI frames. Finally, 2-D strain maps from corresponding echocardiography and tagged MRI were computed and averaged over 17 segments.

Figure 9a shows B-mode and TDI images in early systole at the high papillary muscle level of a subject. The computed motion of the heart between these two frames is shown in Fig. 9b and c based on HS optical flow and TDIOF, respectively. The cardiac strain maps for the same cardiac phase and same slice are shown in Figs. 10 and  11. Figure 11 compares the radial strain map with the tagged MRI radial strain map. As expected and observed from the tagged MRI results, the radial strains from TDIOF are positive and gradually increase during systole. The increased radial strain is more pronounced in AL and IL segments in both SinMod derived and TDIOF strain maps. The increased radial strain is also prominent in the AS and IS segments. Figure 11 shows the circumferential strain map compared to the tagged MRI circumferential strain map. As expected and observed from the tagged MRI results, the circumferential strains from TDIOF are negative and gradually increase in magnitude during systole. This increase is more pronounced in AL and ANT segments in both SinMod tagged MRI-derived and TDIOF strain maps.

Fig. 9

a A short axis B-mode image at the high papillary muscle level of a subject during early systole compared to TDI image for the same phase. b Horn-Schunck motion field for the same phase and in the same subject as a. c Corresponding TDIOF motion field. d Tagged MRI motion field for the same approximate slice location in systole (AS antero-septal, ANT anterior, IL infero-lateral, AL antero-lateral, P posterior, INF inferior, IS infero-septal)

Fig. 10

Top row Lagrangian radial strain maps computed from TDIOF. Lower row Lagrangian radial strain maps computed with SinMod from tagged MRI during the same cardiac phase at the high papillary muscle level in one subject (AS antero-septal, ANT anterior, IL infero-lateral, AL antero-lateral, P posterior, INF inferior, IS infero-septal). The tagged MR images are resized to match the echo images with respect to the size

Fig. 11

Top row Lagrangian circumferential strain maps computed from TDIOF. Lower row Lagrangian circumferential strain maps computed with SinMod from tagged MRI during the same cardiac phase at the high papillary muscle level in one subject (AS antero-septal, ANT anterior, IL infero-lateral, AL antero-lateral, P posterior, INF inferior, IS infero-septal). The tagged MR images have been resized to match the echo images with respect to size

Fig. 12

a TDIOF radial strain value versus tagged MRI radial strain is shown as red; while HS radial strain value versus tagged MRI radial strain value is shown as blue stars; and BM radial strain vs tagged MRI radial strain is shown as green dots. b TDIOF circumferential strain value versus tagged MRI circumferential strain is shown as red; while HS circumferential strain value versus tagged MRI circumferential strain value is shown as blue stars; and BM circumferential strain versus tagged MRI circumferential strain is shown as green dots. For both cases, it is evident that the blue dots (HS strain values) are more scattered compared to the tagged strain values. The plots include corresponding average strain quantities for 17 segments in 8 patients over 4 tagged MRI frames

To compare the performance of TDIOF and HS, statistical analysis of the strain map results are helpful. Figure 12 shows correlation studies of the radial and circumferential strain values compared to tagged MRI. The correlation coefficient (r) for the TDIOF radial strain values compared to the tagged MRI radial strain values was 0.83 ($\mathrm{P}<0.001$); while the correlation coefficient (r) for the HS and BM radial strain values compared to the tagged MRI radial strain values were 0.71($\mathrm{P}<0.001$) and 0.75($\mathrm{P}<0.001$), respectively. The correlation coefficient (r) for the TDIOF circumferential strain values compared to the tagged MRI circumferential strain values was 0.86 ($\mathrm{P}<0.001$); while the correlation coefficient (r) for the HS and BM circumferential strain values compared to the tagged MRI circumferential strain values were 0.77 ($\mathrm{P}<0.001$) and 0.79 ($\mathrm{P}<0.001$). Therefore, it may be concluded that for both radial and circumferential strains, TDIOF analysis achieves a more significant correlation with the tagged MRI in comparison to HS and BM analysis. This effect is believed to be due to the additional Doppler term that is added to the TDIOF framework. The comparison of TDIOF and HS radial strain using student t-test showed superiority of TDIOF (95 % confidence interval, $\mathrm{P}<0.001$). Similarly, the comparison of TDIOF and HS circumferential strain using student t-test showed superiority of TDIOF (95 % confidence interval, $\mathrm{P}<0.001$).The comparison of TDIOF and BM radial strain using student t-test was statistically meaningful (95 % confidence interval, $\mathrm{P}<0.001$).The comparison of TDIOF and BM circumferential strain using student t-test was prominent as well (95 % confidence interval, $\mathrm{P}<0.001$).

6 Discussion

In order to increase the accuracy of motion estimation and speckle tracking techniques and to overcome the angle dependency of TDI, fusion of the techniques has been proposed. TDIOF makes use of the combination of B-mode and Doppler energy terms, minimized using linear algebraic methods. It was demonstrated that TDIOF outperforms the Horn-Schunck optical flow technique and block matching speckle tracking when applied to simulated, physical phantom, and real data. In this paper, we demonstrated that the additional Doppler term is able to increase the accuracy of the intensity (B-mode) based methods in tracking left-ventricular wall motion. The additional Doppler term may very well be added to other cardiac Ultrasound image registration techniques and we expect a corresponding improvement in performance. As demonstrated in the simulation study, the improvement in performance is more pronounced on noisy images.

TDIOF had better performance when compared to HS and Block matching in simulated, phantom, and in vivo data. Due to increased thickness of the wall, the results were better in mid-ventricular slices for all three methods. Nevertheless, results at the basal and apical slices were still acceptable. Due to poor acquisition at the apex, results for apical segments,were not as good for all three techniques compared. Similarly, in comparison to HS and BM, results from TDIOF correlated more significantly with tagged MRI. It is evident from Figs. 11 and 12, that both radial and circumferential strains increase over the cardiac systole, while the heart is contracting and peaks attend systole and then as the heart recoils back to the original length the cardiac strain decreases to about zero at end diastole. It should be noted that the strain values for TDIOF and tagged MRI are not exactly the same because it is not possible to perfectly align the images in space and time due to differences in image slice thickness, resolution, and precise image orientation.

6.1 Comparison with Previous Work

A comparison of correlative strain results for TDIOF reported in this paper can further illustrate the performance of the proposed technique. In [39], a comparison of MRI-derived strains and speckle tracking-derived strains were reported. The authors collected data in patients using a commercially available system (Vivid 7, GE Vingmed Ultrasound AS, Horten, Norway) and performed off-line analysis (EchoPac BT04, GE Vingmed Ultrasound AS). Subsequently, the same group of patients underwent tagged MRI scan and HARP off-line analysis to determine the regional strains. The correlation between radial strain based on B-mode speckle tracking and tagged MRI was reported to be ($\mathrm{r} = 0.60, \mathrm{p} < 0.001$) while the correlation between circumferential and longitudinal strain values based on B-mode speckle tracking and tagged MRI was reported to be ($\mathrm{r} = 0.51, \mathrm{p} < 0.001$) and ($\mathrm{r} = 0.64, \mathrm{p} < 0.001$). The authors concluded that there is a modest correlation between echocardiographic and tagged-MRI-derived strains.

6.2 Limitations

The present study has several limitations that should be stated. At this time it is not possible to extend TDIOF to three dimensions because TDI is only possible in two dimensions.

Another limitation is lack of availability of ground truth applicable to in vivo images which makes the validation more difficult. Tagged MRI is a good surrogate but it is not perfect. Tagged MRI slices do not exactly overlap on echocardiographic slices and there is no accurate pixel to pixel mapping from the cardiac tissue in tagged MRI to the cardiac tissue in echocardiography. Additionally, the orientation of the Ultrasound transducer is not exactly the same as image orientation in tagged MRI. Furthermore, Echocardiography and tagged MRI have different resolutions in space and time.

An additional potential issue is that MRI and echocardiography cannot be performed simultaneously. In our study, since MRI was performed immediately after echocardiography, the cardiac physicologic changes are felt to be less significant. However, heart rate variability may cause alignment problems between the images. We attempted to overcome these issues by careful image acquisition and matching of the slices in space and time.

7 Conclusion

In order to increase the accuracy of the speckle tracking technique and to cope with the angle dependency of TDI, a combined approach dubbed TDIOF (Tissue Doppler Imaging Optical Flow) has been proposed. TDIOF is formulated based on the combination of B-mode and Doppler energy terms minimized using linear algebraic methods. TDIOF was validated extensively based on simulated images, physical cardiac phantom, and in-vivo data. The performance of TDIOF was demonstrated to be better than popular motion estimation and speckle tracking techniques in echocardiography.

Appendices

Appendix A: Mathematical Framework for TDIOF

To determine myocardial motion, we propose a novel optical flow energy function which combines three energy terms: B-mode intensity constancy, Doppler/B-mode velocity similarity, and motion smoothness.

(1) B-mode intensity constancy: If we assume that $p=(x,y,t)$and the flow field is $w(p)=(u(p),v(p),1)$ where $u$ and $v$ are the motion vectors and $x,y$ and $t$ are the spatial and temporal dimensions, B-mode intensity constancy term assumes that the pixel intensity is the same along the motion vector. When $I(p)$ is the pixel intensity at location p and $I(p+w)$ is the pixel intensity in the subsequent frame at location p + w,

$$\begin{aligned} E_{data} =\left| {I(p+w)-I(p)} \right| ^{2} \end{aligned}$$

(A.1)

Although optical flow is usually solved using calculus of variation, we use the recent incremental flow framework [25] which provides significant computational savings. The incremental flow assumes that an estimate of flow is already known (iteration 0) and then, the best increment will be found at each iteration,.With inclusion of an incremental motion vector, the intensity constancy is then revised to be:

$$\begin{aligned} E_{data} =\left| {I(p+w+dw)-I(p)} \right| ^{2} \end{aligned}$$

(A.2)

The above equation can be linearized using Taylor series expansion:

$$\begin{aligned} I_t (p+w+dw)-I(p)=I_t (p)+I_x (p)du(p)+I_y (p)dv(p) \end{aligned}$$

(A.3)

with

$$\begin{aligned} I_x (p)=\frac{\partial I(p+w)}{\partial x} \end{aligned}$$

(A.4)

$$\begin{aligned} I_y (p)=\frac{\partial I(p+w)}{\partial y} \end{aligned}$$

(A.5)

$$\begin{aligned} I_t (p)=I(p+w)-I(p) \end{aligned}$$

(A.6)

(2) The smoothness energy term forces the flow field to be continuous:

$$\begin{aligned} E_s =\left| {\nabla (u+du)} \right| ^{2}+\left| {\nabla (v+dv)} \right| ^{2} \end{aligned}$$

(A.7)

with

$$\begin{aligned} \left| {\nabla (u+du)} \right| ^{2}=\left( {\frac{\partial (u+du)}{\partial x}} \right) ^{2}+\left( {\frac{\partial (u+du)}{\partial y}} \right) ^{2} \end{aligned}$$

(A.8)

(3) TDI velocity term. The 2D motion when projected in the direction of the transducer should be similar to the computed velocity. If $\mathbf{{\vec {v}}}=(u,v)$ is the B-mode velocity, $\mathbf{{\vec {v}}}_t =(u_t ,v_t )$ is the transducer orientation and $w_{tdi}$ is the TDI velocity acquired from the echo machine, then the constraint is formulated as:

$$\begin{aligned} E_{tdi} =\left( \mathbf{{\vec {v}}}^{\mathbf{T}}\mathbf{{\vec {v}}}_\mathbf{t} -w_{tdi} \right) ^{2}=\left( {u_t u+v_t v-w_{tdi} } \right) ^{2} \end{aligned}$$

(A.9)

In order to keep the range of $E_{tdi}$ between 0 and 1, and to reject outliers, we utilized a Geman-Mcclure penalizer ($\psi )$ [26]:

$$\begin{aligned} \psi \mathrm{(s)}=\frac{\mathrm{{s}}^{2}}{\mathrm{{s}}^{2}+{\upsigma }^{2}} \end{aligned}$$

(A.10)

In (A.10), $s$ is the input data and $\sigma$ is the scaling parameter. The behavior of Geman-Mcclure equation is shown in Fig. A.1.

Fig. A.1

Geman-Mcclure penalizer for different $\sigma$ parameters (0.1, 0.3, 0.5, 0.7, 0.9)

The total energy function to be minimized is:

$$\begin{aligned} E_ (u,v)&=E_{data} +\alpha E_s +\beta \psi (E_{tdi} )\nonumber \\&=\int \limits _{\Omega } {(\left| {I(P+w+dw)-I(p)} \right| ^{2}+} \alpha (\left| {\nabla \left( {u+du} \right) } \right| ^{2}+\left| {\nabla \left( {v+dv} \right) } \right| ^{2})\nonumber \\&\qquad \qquad +\beta . \psi ((u_t (u+du)+v_t (v+dv)-w_{tdi} )^{2})) \end{aligned}$$

(A.11)

where $\alpha$ is the smoothness weight and $\beta$ is the TDI/velocity correspondence parameter—we note that setting beta to zero essentially results in the Horn and Schunck optical flow frame case in the incremental flow framework.

Next, we vectorize $u,v,du,dv$ as $U,V,dU,dV$.

$$ {\mathrm{I}}_\mathbf{x} =diag(I_x ) \mathrm{{I}}_\mathbf{y} =diag(I_y ) $$

$\mathrm{{D}}_\mathrm{{x}}$ and $\mathrm{{D}}_\mathrm{{y}}$ are denoted as matrices related to the x and y derivative filters such that: $\mathrm{{D}}_\mathrm{{x}} U=u\otimes [0-1 1]$. The derivative operator is used to compute the gradient of the image in each direction. Additionally the column vector $\delta _p$ is defined as a Dirac function with the only nonzero element at location $p$ such that$\delta _p I_x =I_x (p)$. Now the discretized version of the energy function (Eq. (A.11)) becomes:

$$\begin{aligned} \mathrm{{E}}&=\sum \limits _\mathrm{{p}} (\updelta {^{\!\!\mathrm {T}}_{\!\mathrm {p}}} \mathrm{{(I}}_\mathrm{{t}} +\mathrm{{I}}_\mathrm{{x}} \mathrm{{dU}} +\mathrm{{I}}_\mathrm{{y}} \mathrm{{dV}}))^{2} +\updelta {^{\!\!\mathrm {T}}_{\!\mathrm {p}}}\mathrm {D}_\mathrm{{x}} \mathrm{{(U}}+\mathrm{{dU)}}^{2}+(\updelta {^{\!\!\mathrm {T}}_{\!\mathrm {p}}} \mathrm{{D}}_\mathrm{{y}} \mathrm{(U+dU)}^{2})\nonumber \\&\qquad \qquad +(\updelta {^{\!\!\mathrm {T}}_{\!\mathrm {p}}} \mathrm{{D}}_\mathrm{{x}} \mathrm{(V+dV)}^{2}+(\updelta {^{\!\!\mathrm {T}}_{\!\mathrm {p}}} \mathrm{{D}}_\mathrm{{y}} \mathrm{(V+dV)}^{2})]+\beta ((\updelta {^{\!\!\mathrm {T}}}(\mathrm{u}_\mathrm{{t}} \mathrm{(U+dU)}\nonumber \\&\qquad \qquad +\mathrm{{v}}_\mathrm{{t}} \mathrm{(V+dV)}-\mathrm{{w}}_\mathrm{{tdi}} ))^{2}) \end{aligned}$$

(6)

To minimize (A.12), Iterative Reweighted Least Squares (IRLS) was used with the stopping criterion that $[\frac{\partial E}{\partial dU};\frac{\partial E}{\partial dV}]=0$. Here, it is noteworthy to state that since for matrix $A$ and vectors $x, b$:

$$ \frac{d}{dx}x^{T}Ax=2Ax \frac{d}{dx}x^{T}b=b $$

Therefore:

$$\begin{aligned} \frac{\partial \mathrm{{E}}}{\partial \mathrm{{dU}}}&= 2 \sum \limits _{\mathrm{p}} \left( \mathrm{{I}}_\mathrm{{x}} {\updelta }_{\mathrm{p}} {\updelta }^{\mathrm{T}}_{\mathrm{p}} (\mathrm{I}_\mathrm{{y}} \mathrm{dV}+\mathrm{{I}}_\mathrm{{t}})+\mathrm{{I}}_\mathrm{{x}} {\updelta }_\mathrm{{p}}{\updelta }^\mathrm{{T}}_\mathrm{{p}} \mathrm{{I}}_\mathrm{{x}} \mathrm{{dU}} \right) +{\upalpha } [(\mathrm{D}^\mathrm{{T}}_\mathrm{{x}} {\updelta }_\mathrm{{p}} {\updelta }^\mathrm{{T}}_\mathrm{{p}} \mathrm{{D}}_\mathrm{{x}}\nonumber \\&\qquad \qquad +\mathrm{{D}}^\mathrm{{T}}_\mathrm{{y}} {\updelta }_\mathrm{{p}} {\updelta }^\mathrm{{T}}_\mathrm{{p}} \mathrm{{D}}_\mathrm{{y}} )\mathrm{(U+dU)}] +{\upbeta } {\uppsi }' (\mathrm{E}_\mathrm{{tdi}})[\mathrm{u}_\mathrm{{t}} {\updelta }_\mathrm{{p}} ({\updelta }^\mathrm{{T}}_\mathrm{{p}} \mathrm{((U+dU) u}_\mathrm{{t}}\nonumber \\&\qquad \qquad +(\mathrm{{V}}+\mathrm{{dV}})\mathrm{v}_\mathrm{{t}} -\mathrm{{w}}_\mathrm{{tdi}} ))] =2((\mathrm{I}_\mathrm{{x}}^{2}+\upalpha \mathrm{L}+\upbeta {\uppsi }' \mathrm{u}_\mathrm{{t}}^{2})\mathrm{dU} \nonumber \\&\qquad \qquad +(\mathrm{I}_\mathrm{{x}} \mathrm{{I}}_\mathrm{{y}} +\upbeta {\uppsi }' \mathrm{u}_\mathrm{{t}} \mathrm{{v}}_\mathrm{{t}})\mathrm{dV}+(\alpha \mathrm{{L}}+\upbeta {\uppsi }' \mathrm{u}_\mathrm{{t}}^{2})\mathrm{U}+\upbeta {\uppsi } '\mathrm{u}_\mathrm{{t}} \mathrm{{v}}_\mathrm{{t}} \mathrm{{V}}\nonumber \\&\qquad \qquad +(\mathrm{I}_\mathrm{{x}} \mathrm{{I}}_\mathrm{{t}} -\upbeta {\uppsi } '\mathrm{u}_\mathrm{{t}} \mathrm{{w}}_\mathrm{{tdi}} ) \\ \end{aligned}$$

(A.13)

:

$$\begin{aligned} \frac{\partial \mathrm{{E}}}{\partial \mathrm{{dU}}}&=2((\mathrm{I}_\mathrm{{x}} ^{2}+\upalpha \mathrm{L}+\upbeta {\uppsi }'\mathrm{u}_\mathrm{{t}} ^{2})\mathrm{dU}+(\mathrm{I}_\mathrm{{x}} \mathrm{{I}}_\mathrm{{y}} +\upbeta {\uppsi }'\mathrm{u}_\mathrm{{t}} \mathrm{{v}}_\mathrm{{t}} )\mathrm{dV}\nonumber \\&\quad +(\alpha \mathrm{{L}}+\upbeta {\uppsi }'\mathrm{u}_\mathrm{{t}}^\mathrm{{2}})\mathrm{U}+\upbeta \psi '\mathrm{u}_\mathrm{{t}} \mathrm{{v}}_\mathrm{{t}} \mathrm{{V}}+(\mathrm{I}_\mathrm{{x}} \mathrm{{I}}_\mathrm{{t}} -\upbeta {\uppsi } '\mathrm{u}_\mathrm{{t}} \mathrm{{w}}_\mathrm{{tdi}}) \end{aligned}$$

(A.14)

where:

$$\begin{aligned} \mathrm{{L}}=\mathrm{{D}}_\mathrm{{x}}^\mathrm{{T}}{\uppsi } '\mathrm{D}_\mathrm{{x}} +\mathrm{{D}}_\mathrm{{y}}^\mathrm{{T}}{\uppsi }'\mathrm{D}_\mathrm{{y}} \end{aligned}$$

(A.15)

$$\begin{aligned} {\uppsi }'=\mathrm{{diag}}({\psi }'\mathrm{{(E}}_\mathrm{{tdi}} )) \end{aligned}$$

(A.16)

and $\sum \limits _p {\delta _p ^{T}\delta _p }$ is the identity matrix.

Similarly

$$\begin{aligned} \frac{\partial \mathrm{{E}}}{\partial \mathrm{{dV}}}&=2((\mathrm{I}_\mathrm{{x}} \mathrm{{I}}_\mathrm{{y}} +\upbeta {\uppsi }' \mathrm{u}_\mathrm{{t}} \mathrm{{v}}_\mathrm{{t}} )\mathrm{dU}+(\mathrm{I}_\mathrm{{y}} ^{2}+\upalpha \mathrm{L}+\upbeta {\uppsi }'\mathrm{v}_\mathrm{{t}} ^{2})\mathrm{dV}+\upbeta {\uppsi }' \mathrm{u}_\mathrm{{t}} \mathrm{{v}}_\mathrm{{t}} \mathrm{{U}}+\nonumber \\&\quad +(\alpha \mathrm{{L}}+\upbeta {\uppsi }'\mathrm{v}_\mathrm{{t}} ^{2})\mathrm{V}(\mathrm{I}_\mathrm{{y}} \mathrm{{I}}_\mathrm{{t}} -\upbeta {\uppsi }' \mathrm{v}_\mathrm{{t}} \mathrm{{w}}_\mathrm{{tdi}}) \end{aligned}$$

(A.17)

Finally, the following linear equation is derived as:

$$\begin{aligned}&\left( {{\begin{array}{ll} {\mathrm{{I}}_\mathrm{{x}}^{2}+\upalpha \mathrm{L}+\upbeta {\uppsi }' \mathrm{u}_\mathrm{{t}}^{2}}&{} {\mathrm{{I}}_\mathrm{{x}} \mathrm{{I}}_\mathrm{{y}} +\upbeta {\uppsi }'\mathrm{u}_\mathrm{{t}} \mathrm{{v}}_\mathrm{{t}}} \\ {\mathrm{{I}}_\mathrm{{x}} \mathrm{{I}}_\mathrm{{y}} +\upbeta {\uppsi }' \mathrm{u}_\mathrm{{t}} \mathrm{{v}}_\mathrm{{t}} }&{} {\mathrm{{I}}_\mathrm{{y}}^{2}+\upalpha \mathrm{L}+\upbeta {\uppsi }'\mathrm{v}_\mathrm{{t}}^{2}} \\ \end{array} }} \right) \left( {{\begin{array}{l} {\mathrm{{dU}}} \\ {\mathrm{{dV}}} \\ \end{array} }} \right) \nonumber \\&=\left( {{\begin{array}{ll} {\mathrm{{I}}_\mathrm{{x}} \mathrm{{I}}_\mathrm{{t}} +\upalpha \mathrm{LU}-\upbeta {\uppsi }'\mathrm{u}_\mathrm{{t}} \mathrm{{w}}_\mathrm{{tdi}} +\upbeta {\uppsi }'\mathrm{u}_\mathrm{{t}}^{2}\mathrm{{U}}+\upbeta {\uppsi }'\mathrm{u}_\mathrm{{t}} \mathrm{{v}}_\mathrm{{t}} \mathrm{{V}}} \\ {\mathrm{{I}}_\mathrm{{y}} \mathrm{{I}}_\mathrm{{t}} +\upalpha \mathrm{LV}-\upbeta {\uppsi }' \mathrm{v}_\mathrm{{t}} \mathrm{{w}}_\mathrm{{tdi}} ++\upbeta {\uppsi }' \mathrm{u}_\mathrm{{t}} \mathrm{{v}}_\mathrm{{t}} \mathrm{{U}}+\upbeta {\uppsi }' \mathrm{v}_\mathrm{{t}}^{2}\mathrm{{V}}} \\ \end{array} }} \right) \end{aligned}$$

(A.18)

In practice, $u,v,dU$ and $dV$ are initialized as zero with $dU$ and $dV$iteratively updated using linear least squares. In order to cover a wide range of displacements and to reduce the computational time, the algorithm is applied in a multi-scale strategy. The coarse scale is tackled in the first step, while the fine scale is computed in the last stage.

Appendix B: Block Matching

The main idea in this type of motion estimation is that each block of a frame moves toward a block with similar intensity in the next frame, when the time interval between the two frames is small. The general strategy is to slide each block of the first frame over the next frame in order to locate the most similar match. To find the best matching block, it is necessary to have a similarity metric that measures the similarity between two blocks. There are several well-known block matching algorithms based on different cost functions such as Mean Absolute Difference (MAD), Mean Square Error (MSE), or correlation. MAD is utilized in this paper because of its accuracy and computational efficiency [28]. MAD is formulated as:

$$\begin{aligned} MAD=\frac{1}{N^{2}}\sum _{i=0}^{N-1} {\sum _{j=0}^{N-1} {\left| {C_{ij} -R_{ij} } \right| } } \end{aligned}$$

(A.19)

In (A.9), N is the size of macro-block while $C_{ij}$ and $R_{ij}$ define the pixel locations within the blocks. The index i is the shift in x and j is the shift towards y when the main block is sliding over the image [28].