A Brief Survey on Semi-Lagrangian Schemes for Image Processing

Abstract

In this survey we present some semi-Lagrangian schemes for the approximation of weak solutions of first and second order differential problems related to image processing and computer vision. The general framework is given by the theory of viscosity solutions and, in some cases, of calculus of variations. The schemes proposed here have interesting stability properties for evolutive problems since they allow for large time steps, can deal with degenerate problems and are more accurate if compared to standard finite difference/element methods of the same order. Several examples on classical problems will illustrate these properties.

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Nonlinear Partial Differential Equations (PDEs) appear in many classical image processing problems and there is a need for accurate and efficient numerical methods to solve them. Usually we have to deal with several difficulties, including the fact that in most cases these are degenerate problems with non smooth solutions. Classical methods (Finite Elements, Finite Differences) must be modified and often stabilized to solve this kind of problems (the interested reader will find in the books by Strikwerda [46] and Brenner and Scott [6] a comprehensive introduction to these classical techniques and well as some recent results). The goal of this paper is to present semi-Lagrangian (SL in the sequel) approximation techniques and illustrate their properties through the solution of some classical image processing problems.

SL methods were introduced long time ago for linear advection problems. As far as we know the first example is the Courant-Isaacson and Rees scheme [19] which dates back to the 1960s. However, their application to nonlinear problems is more recent as well as the analysis of their mathematical properties in terms of consistency, stability, convergence and convergence rates. A comprehensive introduction to this class of schemes can be found in the book by Falcone and Ferretti [32].

Let us just recall here some of the motivations which make semi-Lagrangian schemes an interesting alternative to more classical finite differences and finite elements techniques. The first and perhaps more interesting point is that, due to the fact that they mimic the method of characteristics, they allow for large time steps in evolutive problems. This property is particularly useful when one looks for the limiting behavior for t going to infinity in order to recover the solution of the corresponding stationary problem. The second important property is that they introduce only a small numerical viscosity so that their smoothing effect on the solution is very limited. The third property to be mentioned is that they have a natural interpretation in terms of the representation formula for some nonlinear problems. One example is the solution of the eikonal equation where the Hopf-Lax formula applies (see the next section for details). Roughly speaking they can be interpreted as a discrete Hopf-Lax formula and this means that their accuracy can be increased to high-order. Finally, they can work on structured and unstructured grids. Although the latter are not very popular among the image processing community they are a natural choice in other fields of applications like fluid dynamics, meteorology, geophysics. The details regarding the analytical properties of the schemes will not be illustrated here since this survey is mainly intended to present the schemes and their potential via the solution of some test problems.

In the next section we will briefly describe how the schemes are derived for some first and second order problems which come from image processing. We will consider some simplified models in order to introduce the main ideas of SL approximation and to explain how the methods are constructed. The following sections are devoted to some classical problems in image processing: Shape from Shading, nonlinear filtering, segmentation and optical flow. For each of them we will introduce the model which we are going to approximate, present the corresponding SL scheme and illustrate some numerical experiments.

2 An Introduction to Semi-Lagrangian Schemes for Nonlinear PDEs

We will consider three classical examples to show the main features and properties of the SL approximation: Shape from Shading (SfS in the sequel), segmentation via the level set (LS) method and nonlinear filtering via Mean Curvature Motion (MCM). It is worth to note that they all lead to nonlinear (and possibly degenerate) partial differential models.

Let us start with the Shape from Shading problem. This is an inverse problem where we want to compute the surface corresponding to a single image. A simple model for this problem can be derived (see the next section for more details) assuming that the light source is at infinity in the direction ω, the surface is Lambertian, and that we can neglect perspective deformations. We consider then the equation which appears in most of the papers and corresponds to frontal light source at infinity, i.e. ω = (0, 0, 1) and

$$\left \vert \nabla u(x)\right \vert = f(x)\qquad \mbox{ for }x \in \Omega ,$$

(9.1)

where $I : \Omega \subset {\mathbb{R}}^{2} \rightarrow [0,1]$ represents the brightness function of the image and

$$f(x) = \sqrt{ \frac{1} {I{(x)}^{2}} - 1}.$$

(9.2)

This is an eikonal equation, a first order stationary nonlinear pde. In order to solve (9.1) we must provide some boundary conditions on ∂Ω and/or add some informations to select a unique solution. In fact, the equation just depends on the gradient so that if u is a solution also u + constant is a solution and we need to fix the height of the surface at least at the boundary to select a solution. For an image containing an “occluding boundary”, it is usual to consider this boundary as ∂Ω and the domain of reconstruction will be the region Ω enclosed by the occluding boundary.

Another classical problem is the segmentation problem, where we want to detect the boundaries of objects represented in a picture. A very popular method for segmentation is based on the level set method, this application is often called “active contour” since the segmentation is obtained following the evolution of a simple curve (a circle for example) in its normal direction. Starting from a circle inside an object we follow its evolution until the curve touches the borders of the object. It is important to know that the LS method has had a great success for the analysis of front propagation problems for its capability to handle many different physical phenomena within the same theoretical framework. Moreover, LS methods allow to develop the analysis of interface propagation also after the onset of singularities (as for example, when there is a topological change in the front). See the books by Sethian [45] and Osher and Fedkiw [42].

In the LS methods the unknown is a “representation” function $u\,:\,{\mathbb{R}}^{2}\,\times \,[0,T]\,\rightarrow \,\mathbb{R}$ of the interface, the position of the interface Γ _t at time t is given by the 0-level set of u(. , t), i.e.

$$\Gamma _{t} :=\{ x : u(x,t) = 0\}.$$

The model equation corresponding to the LS method (see e.g.[43] for details) is

$$\left \{\begin{array}{l@{\quad }l} u_{t} + c(x,t)\vert \nabla u\vert = 0\quad &(x,t) \in {\mathbb{R}}^{2} \times [0,T] \\ u(x) = u_{0}(x) \quad &x \in {\mathbb{R}}^{2}\end{array} \right.$$

(9.3)

where u ₀ must be a representation function for the initial front∂Ω ₀, where Ω ₀ is a given open bounded subset of ${\mathbb{R}}^{2}$, this means

$$\left \{\begin{array}{l@{\quad }l} u_{0}(x) > 0\quad &\mbox{ for }x \in {\mathbb{R}}^{2} \setminus \overline{\Omega }_{0} \\ u_{0}(x) = 0\quad &\mbox{ for }x \in \partial \Omega _{0} := \Gamma _{0} \\ u_{0}(x) < 0\quad &\mbox{ for }x \in \Omega _{0}.\\ \quad \end{array} \right.$$

(9.4)

Note that usually the scalar velocity $c : {\mathbb{R}}^{2} \times [0,T] \rightarrow \mathbb{R}$ is given and it represents the velocity of the front in its normal direction $\eta (x,t) := \frac{\nabla u(x,t)} {\vert \nabla u(x,t)\vert }$. However, the same approach applies to other scalar velocities. For example one can use it to describe isotropic and anisotropic front propagation, Mean Curvature Motion (MCM) and other situations when the velocity depends on some geometrical properties of the front. In summary, this corresponds to the following choices:

$$\displaystyle\begin{array}{rcl} c(x,t)\text{isotropic growth with time varying velocity}& &\end{array}$$

(9.5)

$$\displaystyle\begin{array}{rcl} c(x,t,\eta )\text{anisotropic growth, i.e. dependent on the normal direction}& &\end{array}$$

(9.6)

$$\displaystyle\begin{array}{rcl} c(x,t,k)\mbox{ Mean Curvature Motion}& &\end{array}$$

(9.7)

where k = k(x, t) is the mean curvature to the front at time t. There are even models where the velocity is obtained by convolution (nonlocal velocity).

Since the borders of objects inside a given image I are characterized by the fact that there is a rapid change in the values of I, in the segmentation problem one can take the gradient of I as an indicator of the edges. If | ∇ I | is large this means that we are at an edge. For this reason, a popular choice of the velocity when we apply LS methods to the segmentation problem is

$$c(x) := {(1 + \vert \nabla I(x){\vert }^{p})}^{-1}\mbox{ where }p \geq 1.$$

With this choice, the velocity is 1 inside the region where the gray level is constant and rapidly decreases to 0 when x is close to an edge (which corresponds to a jump in the I derivative).

Another problem we will consider is nonlinear filtering. A classical model to treat this problem is to consider (9.3) with the velocity given by Mean Curvature c(x, t) = k(x, t), see [34]. In this case one gets

$$\left \{\begin{array}{ll} u_{t}(x,t)& = \mathrm{div}\left ( \dfrac{\nabla u(x,t)} {\vert \nabla u(x,t)\vert }\right )\vert \nabla u(x,t)\vert \quad (x,t) \in {\mathbb{R}}^{2} \times [0,T] \\ u(x,0) & = I_{0}(x) \end{array} \right.$$

(9.8)

where I ₀ is a starting noisy image. This equation, in the level set formalism, represents the Mean Curvature Motion of curves, since each level set of u moves in normal direction with a speed proportional to their mean curvature. The goal is to recover a new image reducing the noise. Note that the evolutive equation is not well defined at all points where the gradient vanishes so it is a singular parabolic equation. This implies that classical finite difference and finite element methods must be adapted in order to deal with the singularities (see [43] for a finite difference scheme and [15, 41] for two convergence results).

Let us now turn our attention to the construction of the semi-Lagrangian schemes for the above model equations. For readers’ convenience, we will start from the linear advection equation:

$$u_{t} + b(x,t) \cdot \nabla u = f(x,t)\quad \quad (x,t) \in {\mathbb{R}}^{2} \times (0,T).$$

(9.9)

Here, $b : {\mathbb{R}}^{2} \times (0,T) \rightarrow {\mathbb{R}}^{2}$ is a vectorfield (the drift) and $f : {\mathbb{R}}^{2} \times (0,T) \rightarrow \mathbb{R}$ is the source term. We look for the unique solution $u : {\mathbb{R}}^{2} \times (0,T) \rightarrow \mathbb{R}$ of (9.9) satisfying the initial condition

$$u(x,0) = u_{0}(x)\quad \quad x \in {\mathbb{R}}^{2}.$$

(9.10)

A simple case corresponds to f(x, t) ≡ 0 and b(x, t) ≡ c (constant). Then, the solution u is given by the representation formula

$$u(x,t) = u_{0}(x - c\;t)\quad \quad (x,t) \in {\mathbb{R}}^{2} \times [0,T)$$

(9.11)

which can be easily obtained by the method of characteristics. The representation formula (9.11) contains an important information: the solution at the point x at time t only depends on the initial value at the point x − ct which is called the foot of the characteristic passing through (x, t) (more details on the method of characteristics can be found on many classical books presenting the basic theory of partial differential equations, e.g. [28]). A typical choice is to build a space grid with constant step Δx

$$\mathcal{G}_{\Delta x} :=\{ x_{j} : x_{j} = (j_{1}\Delta x,j_{2}\Delta x)\mbox{ for }j_{1} \in \mathbb{Z},\;j_{2} \in \mathbb{Z}\}.$$

(9.12)

This grid is called lattice. In order to build an approximation scheme for (9.9) and more in general for evolutive problems, we need to consider a space-time grid

$$\displaystyle\begin{array}{rcl} \mathcal{G}_{\Delta x,\Delta t} :=\{ (x_{j},t_{n}) : x_{j} = (j_{1}\Delta x,j_{2}\Delta x),\;t_{n} = n\Delta t,\mbox{ for }j_{1} \in \mathbb{Z},\;j_{2} \in \mathbb{Z},\;n \in \mathbb{N}\}& &\end{array}$$

(9.13)

where in the simplest case Δt is a constant time step. Naturally for numerical purposes the number of nodes of our grids has to be finite and we will consider the intersections of our grids with the computational domain Ω.

The basic idea behind all finite difference approximations is to replace every derivative by an incremental ratio. Thus, one obtains a finite dimensional problem whose unknown are the values of the numerical solution at all the nodes of the lattice, so that the value $u_{j}^{n}$ associated to the node $(x_{j},t_{n})$ should be regarded as an approximation of $u(x_{j},t_{n})$. For the time derivative it is natural to choose the forward incremental ratio

$$u_{t}(x,t) \approx \frac{u(x,t + \Delta t) - u(x,t)} {\Delta t}.$$

(9.14)

For finite difference, it is well known that dealing with advection problems we need to consider the incremental ratio in the up-wind direction, so that we must take the first node on the left (right) if the velocity is positive (negative). This choice guarantees the stability of the approximation scheme as well as a good fit with the physics behind the model. When $b(x_{j},t_{n})$ is positive, this corresponds to the following up-wind scheme for the left hand side of the advection equation

$$\displaystyle\begin{array}{rcl} & & u_{t}(x_{j},t_{n}) + b(x_{j},t_{n}) \cdot \nabla u(x_{j},t_{n}) \approx \frac{u(x_{j},t_{n} + \Delta t) - u(x_{j},t_{n})} {\Delta t} + \\ & & \quad - b_{1}(x_{j},t_{n})\frac{u(x_{j_{1}-1,j_{2}},t_{n})\,-\,u(x_{j_{1},j_{2}},t_{n})} {\Delta x} \,-\,b_{2}(x_{j},t_{n})\frac{u(x_{j_{1},j_{2}-1},t_{n})\,-\,u(x_{j_{1},j_{2}},t_{n})} {\Delta x}.\end{array}$$

(9.15)

Adopting the standard notation $u_{j}^{n} = u(x_{j},t_{n})$, we write the scheme in the standard explicit time marching form

$$u_{j}^{n+1} = u_{ j}^{n} + \frac{\Delta t} {\Delta x}b(x_{j},t_{n}) \cdot (u_{j_{1}-1,j_{2}}^{n} - u_{ j}^{n},u_{ j_{1},j_{2}-1}^{n} - u_{ j}^{n}).$$

(9.16)

Semi-Lagrangian schemes are based on a different way to construct the approximation of (9.9), in particular this affects the discretization of the advection term b(x, t) ⋅ ∇ u. Since this is a directional derivative, we can write

$$\displaystyle\begin{array}{rcl} b(x_{j},t_{n}) \cdot \nabla u(x_{j},t_{n}) \approx -\frac{{u}^{n}(x_{j} -\delta b(x_{j},t_{n})) - u_{j}^{n}} {\delta } ,& &\end{array}$$

(9.17)

where δ is a “small” positive parameter, and  u ⁿ denotes an extension of the numerical solution (at time t _n ) to be computed outside of the grid. Coupling the forward finite difference in time with this approximation we get

$$\frac{u_{j}^{n+1} - u_{j}^{n}} {\Delta t} -\frac{{u}^{n}(x_{j} -\delta b(x_{j},t_{n})) - u_{j}^{n}} {\delta } = 0,$$

(9.18)

and finally, choosing δ = Δt, we obtain the scheme

$$u_{j}^{n+1} = {u}^{n}(x_{ j} - \Delta tb(x_{j},t_{n})).$$

(9.19)

This is the semi-Lagrangian scheme for (9.9) for f(x, t) = 0. Note that, for b(x, t) ≡ c (9.18) can be easily interpreted as a discrete version of the representation formula (9.11). Even in this simple case, the value at the point x _j  − Δtc does not belong to the grid $\mathcal{G}_{\Delta x}$ and must be obtained by interpolation on the values at the grid nodes. We will denote by Π[w] the interpolation operator which reconstructs locally the values of a function w only known on $\mathcal{G}_{\Delta x}$. There are many possible choices for Π, e.g. linear, bilinear or cubic interpolation (see also [32] for more general choices). A very popular choice is the piecewise linear interpolation which produces a monotone first order scheme.

Now let turn our attention to the nonlinear case and consider the eikonal equation. First note that

$$\vert \nabla u(x)\vert =\max _{a\in B(0,1)}a \cdot \nabla u(x).$$

(9.20)

where B(0, 1) denotes the unit ball centered at the origin. In fact, the maximum in (9.20) is attained at ${a}^{{\ast}} = \nabla u(x)/\vert \nabla u(x)\vert$ and this shows that the right-hand side coincides with | ∇ u(x) | . Using now (9.20) to rewrite Eq. (9.3), we get

$$u_{t} +\max _{a\in B(0,1)}[c(x,t)a \cdot \nabla u(x)] = 0.$$

(9.21)

Since the term inside the square brackets is linear, we can apply the semi-Lagrangian scheme for the advection equation obtaining the following scheme

$$u_{j}^{n+1} =\min _{ a\in B(0,1)}[{u}^{n}(x_{ j} - \Delta tc(x_{j},t_{n})a)].$$

(9.22)

Note that again u ⁿ is computed at a point which in general is not a node of the grid, this will require a local reconstruction via the interpolation operator Π. We can write (9.22) in a more explicit way as

$$u_{j}^{n+1} =\min _{ a\in B(0,1)}[\Pi [{u}^{n}](x_{ j} - \Delta tc(x_{j},t_{n})a)].$$

(9.23)

The meaning of the right-hand side is: to recover the correct solution of the nonlinear stationary equation one needs to solve a family of advection problems corresponding to a vector field pointing in every direction and then has to take the minimum value among all possible ones. This step clearly requires a constrained optimization method to be solved. In practice, for low order accuracy it suffices to compare a finite number of values corresponding to a uniform discretization of the unit ball.

It is important to note that the SL scheme is up-wind by construction and this explains its strong stability properties which allow to use large time steps with respect to other methods.

The derivation of an efficient discretization scheme for the MCM equation is more complicated. One can think to simply substitute $c_{j}^{n} = k_{j}^{n}$ where $k_{j}^{n} = k(x_{j},t_{n})$ is the curvature at the point x _j at time t _n . This can be done using a discrete version of the curvature for the level set of the solution as in [43]. However, via a stochastic interpretation of the equation, one can derive a semi-Lagrangian approximation where the role of characteristics is taken by the solution of an appropriate stochastic differential equation associated to the equation. More information on this approach for the MCM can be found in [14] and are too technical to be presented here. However, we will present the resulting SL scheme in Sect. 9.4.1.

In conclusion, the construction of SL-schemes is based on three essential building blocks: an integration backward along the characteristics (or the generalized characteristics in the second order problem), an interpolation to get the values at the foot of the characteristics and an optimization method to recover the minimum among all the values. The first step relies on standard techniques for ordinary differential equation, the interpolation can be obtained either by standard polynomial techniques or via more sophisticated techniques, for example ENO (Essentially Non Oscillatory) or WENO (Weighted ENO) methods as in [13]. Finally, for the optimization step, one can simply discretize the unit ball or use a minimization method without derivatives such as Brent algorithm [7]. An extensive presentation of those building blocks and an analysis of SL schemes can be found in [32].

3 Shape from Shading

As we said, we want to reconstruct a surface u(x) from a single image.

This means that given an image I we are looking for the surface u(x) corresponding to the object represented in it. The irradiance function I is the datum in the model since it is measured at each pixel $x := (x_{1},x_{2})$ of the image, for example in terms of a grey level (from 0 to 255). To construct a continuous model, we will assume that I takes real values in the interval [0, 1]. The height function u which is the unknown of the problem has to be reconstructed on the “reconstruction domain” Ω. The characterization of the surface via a partial differential equation relies on several assumptions. Assume that there is a unique light source at infinity whose direction is indicated by the unit vector $\omega = (\omega _{1},\omega _{2},\omega _{3}) \in {\mathbb{R}}^{3}$. Also assume for simplicity that ω is given. For a Lambertian surface of uniform albedo equal to 1, the reflectance function giving the brightness at a point on the surface is R(η(x)) : = ω ⋅η(x), where η(x) is the unit normal to the surface at $(x_{1},x_{2},u(x))$. This equation is called the irradiance equation. Recalling that the normal is given by $\eta (x) = (-u_{x_{1}},-u_{x_{2}},1)$ we obtain the following equation

$$I(x) = \frac{(\omega _{1},\omega _{2}) \cdot \nabla u(x) -\omega _{3}} {\sqrt{1 + \vert \nabla u(x){\vert }^{2}}} \qquad \mbox{ for }x \in \Omega ,$$

(9.24)

which is a first order nonlinear partial differential equation of Hamilton-Jacobi type. We have seen that in the vertical light source case (i.e. ω = (0, 0, 1)) this reduces to the eikonal equation (9.1).

As we said, this equation must be complemented with boundary conditions. A natural choice is to consider homogeneous Dirichlet type boundary conditions corresponding to flat background i.e., setting

$$u(x) = 0\qquad \mbox{ for }x \in \partial \Omega.$$

(9.25)

However, one can also choose a more general Dirichlet boundary condition

$$u(x) = g(x)\qquad \mbox{ for }x \in \partial \Omega.$$

(9.26)

The solution of the above Dirichlet problems (9.24) and (9.25) or (9.24) and (9.26) will give the surface corresponding to grey level I(x) measured in Ω. To set u(x) = g(x) at least on a part of the boundary we must know something about the surface. This is the case for surfaces obtained by revolution around an axis (as for our vase in Fig. 9.1).

Fig. 9.1

The image and its corresponding surface (approximation)

In order to illustrate one of the features of the SL scheme, let us consider a virtual image corresponding to the surface

$$u(x_{1},x_{2}) = \left \{\begin{array}{l@{\quad }l} 2(1 -\vert x_{1}\vert )\quad &x_{1} \in [-1,1],\quad x_{2} \in \left [-1,-\dfrac{1} {2}\vert x_{1}\vert -\dfrac{1} {2}\right ] \\ 2(1 -\vert x_{2}\vert )\quad &x_{1} \in [-1,1],\quad x_{2} \in \left [\dfrac{1} {2}\vert x_{1}\vert + \dfrac{1} {2},1\right ] \\ 1 -\vert x_{1}\vert \quad &\qquad \qquad \qquad \mathrm{otherwise}. \end{array} \right.$$

The surface looks like a “ridge tent” (see Fig. 9.2), so it is non regular but the boundary conditions are very simple: 0 on each side of the rectangle. We apply our SL scheme for the eikonal equation (9.22). Looking at the right picture of Fig. 9.2, we can see that the algorithm is accurate around the kinks and that there are no spurious oscillations where the gradient is discontinuous. Similar results have been obtained for other virtual and real images. A detailed presentation of these results as well as a comparison with other numerical methods can be found in the survey [27].

Fig. 9.2

(a) The original surface, (b) its brightness function in the plane, (c) the approximate surface

Let us also mention that the same problem can be solved by applying optimization methods based on the variational approach. The interested reader can find in Horn and Brooks’ book [35] several results and references relating to the variational approach. In this class of methods two basic ingredients must be chosen: the functional which has to be optimized (in fact, minimized) and the minimization method. The first difficulty encountered in the SfS problem is the choice of unknowns. The natural unknown is of course height u as we have done before. However, this is not the only choice since u appears in the image irradiance equation only through its first derivatives $p = \partial u/\partial x_{1}$ and $q = \partial u/\partial x_{2}$, which are two non-independent functions. In fact, for u ∈ C ² we know that

$$\partial p/\partial x_{2} = \partial q/\partial x_{1}.$$

(9.27)

The only technical difficulty with these unknowns is that p or q becomes infinite at each point x belonging to an occluding boundary. This is not a real trouble if no point x in the reconstruction domain Ω is such that I(x) = 0. As equation (9.27) is a hard constraint on p and q, the most natural functional associated with the irradiance equation and (9.27) is

$$\displaystyle\begin{array}{rcl} \mathcal{F}_{1}\left (p,q,\mu \right ) =\displaystyle\int _{\Omega }{\left [r(p(x),q(x)) - I(x)\right ]}^{2}\,\mathit{dx} +\displaystyle\int _{ \Omega }\mu (x)\left [ \frac{\partial p} {\partial x_{2}}(x) - \frac{\partial q} {\partial x_{1}}(x)\right ]\,\mathit{dx}& &\end{array}$$

(9.28)

where μ is a Lagrange multiplier and the function r is such that r(p(x), q(x)) = R(n(x)). Note that $\mathcal{F}_{1}$ is defined on the unknowns p, q and μ. However Horn and Brooks have shown that the three Euler equations associated with $\mathcal{F}_{1}$ can be reduced, for u ∈ C ², to the Euler equation associated with the following functional:

$$\mathcal{F}_{2}\left (p,q\right ) =\displaystyle\int _{\Omega }{\left [r\left (p,q\right ) - I(x)\right ]}^{2}\,\mathit{dx}.$$

(9.29)

The choice of p and q as unknown is due to the fact that the algorithms dealing directly with u are very slow, this choice implies that at the end we will need also another procedure to get back to u. Typically this is done integrating along paths.

As we said, in the previous model we have neglected the perspective deformations. New models have been studied to deal with this problem, which is usually called in the literature Perspective Shape from Shading (PSfS in short) problem. Let us define our model problem adopting the same notation used in [20] (a different model with an attenuation term has been studied in [44]). The point $(X_{0},Y _{0})$ is the principal point of the image, i.e. the intersection between the optical axis of the camera and the perspective plane Π (the film). The parameters d and d′ are respectively the distance of the optical lens from the perspective plane and the distance of the optical lens from the (flat) background, l and $l^\prime = \frac{d^\prime} {d}l$ are respectively the length of a segment in the perspective plane (i.e. in the image) and the length of the real segment corresponding to it (see Fig. 9.3 for more details). The representation of the surface defined by points with coordinates (x, y, z) in terms of the (X, Y ) coordinates of the points in the perspective plane is given by three parametric equations

$$x = r(X,Y ),\;y = s(X,Y ),\;z = t(X,Y )$$

(9.30)

where (see [20])

$$\left \{\begin{array}{ll} r(X,Y ) = \dfrac{X - X_{0}} {d} t(X,Y ) \\ s(X,Y ) = \dfrac{Y - Y _{0}} {d} t(X,Y ). \end{array} \right.$$

(9.31)

The problem is then reduced to compute the third component t. Writing the classical irradiance equation, taking into account the new variables for the perspective deformation and introducing the new variable

$$\overline{t}(X,Y ) := t(X,Y ) + (X - X_{0},Y - Y _{0}) \cdot \nabla t(X,Y ),$$

(9.32)

we can get the following eikonal type equation

$${ \left [ \frac{d} {\overline{t}(X,Y )}\right ]}^{2}\vert \nabla t(X,Y ){\vert }^{2} = \frac{I_{max}^{2}} {I^\prime{(X,Y )}^{2}} - 1\quad \mbox{ in }\Omega $$

(9.33)

Fig. 9.3

Diagram of the optical lens and of the perspective transformation

where

$$\displaystyle\begin{array}{rcl} I^\prime(X,Y ) := \frac{I(X,Y )} {{\mathit{cos}}^{4}\alpha (X,Y )},& &\end{array}$$

(9.34)

I _max is a constant depending on the parameters of the problem (typically is the maximum value of I measured on the image) and the α is the angle between the optical axis and the the ray starting at a point of the scene and passing through the optical center. Note that α varies with the point. Moreover, the coefficient

$${ \cos }^{4}(\alpha (X,Y )) = \frac{{d}^{4}} {{({(X - X_{0})}^{2} + {(Y - Y _{0})}^{2} + {d}^{2})}^{2}},$$

(9.35)

is used to decrease the brightness of the points corresponding to a large α. Note that Ω is the internal region bounded by the silhouette of the object (∂Ω will denote its boundary) which is embedded in a rectangular domain Q, so that the set Q ∖ Ω is the background.

Defining

$$\displaystyle\begin{array}{rcl} f(X,Y ) := \frac{1} {{d}^{2}}\left ( \frac{I_{max}^{2}} {I^\prime{(X,Y )}^{2}} - 1\right )& &\end{array}$$

(9.36)

we can write (9.33) in short as

$$\vert \nabla t(X,Y )\vert = \sqrt{f(X, Y )}\ \Big{\vert }\overline{t}(X,Y )\Big{\vert }.$$

(9.37)

We want to write (9.37) in a fixed point form and construct an approximation scheme for this equation. To this end it is important to note that $\bar{t}$ has a sign. In fact, the normal to the original surface in the point P is given by

$$\widehat{n}(P) = N(P)/\vert N(P)\vert $$

(9.38)

where

$$\displaystyle\begin{array}{rcl} N(P) \equiv \left ( \frac{\partial t} {\partial X}(X,Y ), \frac{\partial t} {\partial Y }(X,Y ),-\bar{t}(X,Y )\right )& &\end{array}$$

(9.39)

and since $-\bar{t}$ must be positive (according to the orientation of the z axis in Fig. 9.3), $\bar{t}$ must be negative. This implies that (9.37) is in fact

$$\vert \nabla t(X,Y )\vert + \sqrt{f(X, Y )}(t(X,Y ) + (X - X_{0},Y - Y _{0}) \cdot \nabla t(X,Y )) = 0$$

which can be written in short as

$$H((X,Y ),t,\nabla t) = 0,\quad \mbox{ in }\Omega $$

(9.40)

where the Hamiltonian H represents the left-hand side of (9.40). It should be noted that, with respect to the classical SfS problem without perspective, we have an additional term which is linear in ∇ t.

Let us consider equation (9.40) complemented with the Dirichlet boundary condition

$$t = g(X,Y )\quad \mbox{ on }\partial \Omega ,\quad \mbox{ where } - d^\prime \leq g \leq 0$$

(9.41)

The arguments presented in Sect. 9.2 for the approximation of the directional derivative lead to the standard semi-Lagrangian scheme for (9.40) which we write in a fixed point form as

$$t(X,Y ) = F[t](X,Y ),\quad \mbox{ in }\Omega $$

(9.42)

where the operator F is defined as

$$F[t](X,Y ) := \frac{1} {1+\delta }\inf _{a\in B(0,1)}\left \{t\left ((X,Y ) +\delta \left ( \frac{-a} {\sqrt{f}} - (X,Y )\right )\right )\right \},\,\mbox{ on }\Omega.$$

(9.43)

and B(0, 1) is the unit ball in ${\mathbb{R}}^{2}$. Introducing a space discretization, and defining $x_{j} = (X_{j_{1}},Y _{j_{2}})$, we can write for every node $x_{j} \in \Omega \cap \mathcal{G}_{\Delta x}$ Eq. (9.42)

$$t(x_{j}) = F[t](x_{j}) := \frac{1} {1+\delta }\inf _{a\in B(0,1)}\left \{\Pi [t]\left (x_{j} +\delta \left ( \frac{-a} {\sqrt{f(x_{j } )}} - x_{j}\right )\right )\right \}$$

(9.44)

which gives the fully discrete semi-Lagrangian scheme for the PSfS problem. Naturally, this equation must be complemented with the boundary conditions (9.41).

In the following test we start from a real photograph where the effect of perspective is visible. The surface is a sheet of paper with the shape of a roof tile. For this image the parameter values are: l = 6. 91 mm, d = 5. 8 mm, l′ = 200 mm, $d^\prime\,=\,\frac{l^\prime} {l}d\,=\,167.87$ mm, Δx = 0. 05 mm. We note that we performed a light correction in the preprocessing step, so we can assume I _max  = 1 during computation. Figure 9.4 shows the photograph (128 ×128 pixels) and the surface reconstructed using Dirichlet boundary conditions (t = 0 on the left and right sides of the boundary and t = g on the top and the bottom). We can see that the solution is quite good considering the fact that light source (flash camera) is not far from the object and that direction of light source is not perfectly vertical as the mathematical model would have required.

Fig. 9.4

A real photograph of one page, 128 × 128 pixels (left) and reconstructed surface with Dirichlet boundary conditions (right)

4 Nonlinear Filtering via MCM

The aim of this section is to present two classical methods based on Mean Curvature Motion for image restoration. These models can be written in the following general form:

$$\left \{\begin{array}{@{}l@{\quad }l@{}} u_{t}(x,t) =\mathrm{ div}{\left ( \dfrac{\nabla u(x,t)} {\vert \nabla u(x,t)\vert }\right )}^{\beta }\vert \nabla u(x,t)\vert \quad \quad &(x,t) \in \Omega \times (0,T] \\ u(x,0) = I_{0}(x)\quad \quad &x \in \Omega \\ \dfrac{\partial u(x,t)} {\partial \eta } = 0\quad \quad &(x,t) \in \partial \Omega \times (0,T] \end{array} \right.$$

(9.45)

where η(x) is the normal direction at the point x. We will consider the case β = 1 corresponding to classical Mean Curvature Motion (MCM) and the case $\beta = 1/3$ corresponding to Affine Morphological Scale Space (AMSS). Here {u(x, t)} _t ≥ 0 represents a family of successive restored versions of the initial noisy image I ₀. The variable t is called the scale variable and, as t increases, we expect u(x, t) be a smoothed image, in which the noise has been removed. The model is able to preserve significant features, like edges and corners, which are typically lost when using linear PDE-methods based on the heat equation where the smoothing effect is isotropic. In curvature-related diffusion, the smoothing effect is anisotropic, i.e. the diffusion depends on the image and is not constant in all directions. The model with β = 1 corresponds to “pure” anisotropic diffusion and it was introduced for image processing by Alvarez et al. in [1]. The term $\mathrm{div}\left ( \frac{\nabla u(x,t)} {\vert \nabla u(x,t)\vert }\right )\vert \nabla u(x,t)\vert$ represents a degenerate diffusion term, which diffuses in the direction orthogonal to the gradient ∇ u and does not diffuse in the direction of ∇ u. In fact, denoting by $\sigma (\nabla u) ={ \left ( \frac{\nabla u} {\vert \nabla u\vert }\right )}^{\perp }$ the unit vector orthogonal to ∇ u, a formulation of the previous equation with respect to σ( ∇ u) would be:

$$u_{t} =\sigma {(\nabla u)}^{\top }{D}^{2}u\sigma (\nabla u),$$

where D ² u is the Hessian matrix with respect to x. This equation admits a unique continuous viscosity solution, if u(x, 0) is bounded uniformly continuous, see [18, 29].

In the paper [2] by Alvarez et al., a connection between scale space analysis and PDEs is proved, the authors also show that the MCM operator satisfies mostly all the relevant properties in multi-scale analysis as monotonicity, invariance under grey scale change, translation invariance. Moreover, they prove that the only operator satisfying all these properties together with affine invariance is the model represented by (9.45) with $\beta = 1/3$. Affine invariance requires the solution of the equation to be invariant with respect any affine transformation: let $\{T_{t}\}_{t\geq 0}$ represents a family of operators which applied to the original image yields a sequence of images u(x, t) = T _t (u), solution of (9.45), then the affine invariance is stated as follows:

$$T_{t}(u\circ \phi ) = T_{t\cdot \vert J_{\phi }\vert }u\circ \phi$$

for any affine map ϕ, where J _ϕ represents the Jacobian of ϕ, which is an invertible matrix. Such property guarantees that shapes are better maintained during smoothing, a classical example is the evolution of an ellipse. It is well known that under MCM any convex curve will propagate to a circle and then collapse to point, instead by the AMSS model the ellipse will collapse to a point preserving the shape, we show the evolution of an ellipse by MCM and AMSS in Fig. 9.5. This property makes the AMSS model particularly attractive for image analysis. In next section, we will show a semi-Lagrangian approximation of both equations with an application to image denoising.

Fig. 9.5

Shrinking of an ellipse by MCM (left) and AMSS (right)

4.1 SL Approximation for the Nonlinear Filtering Problem via MCM

We present a semi-Lagrangian (SL) scheme to approximate equation (9.45) for MCM, case β = 1, and AMSS, case $\beta = 1/3$. For the first equation, we use the scheme described in [14], where a fully explicit scheme is obtained. We denote by $u_{j}^{n}$ the numerical approximation of u(x, t) at the node $(x_{j},t_{n}) \in \mathcal{G}_{\Delta x,\Delta t}$, by $D_{j}^{n}$ the central finite difference approximation of the space gradient, by $\mathcal{D}(j) =\{ (j_{1} + 1,j_{2}),(j_{1} - 1,j_{2}),(j_{1},j_{2} + 1),(j_{1},j_{2} - 1)\}$ and we define $\sigma _{j}^{n} =\sigma (D_{j}^{n})$.

Now, we can write the SL scheme for MCM:

$$u_{j}^{n+1} \equiv \left \{\begin{array}{@{}l@{\quad }l@{}} \frac{1} {2}\left (\Pi [{u}^{n}](x_{ j} +\sigma _{ j}^{n}\sqrt{\Delta t}) + \Pi [{u}^{n}](x_{ j} -\sigma _{j}^{n}\sqrt{\Delta t})\right )\quad &\mbox{ if }\vert D_{ j}^{n}\vert > C\Delta {x}^{s} \\ \frac{1} {4}\mathop{\displaystyle\sum }\limits_{i \in \mathcal{D}(j)}u_{i}^{n} \quad &\mbox{ if }\vert D_{ j}^{n}\vert \leq C\Delta {x}^{s}. \end{array} \right.$$

(9.46)

Here, C and s are positive constant. Let us notice that the vector $\sigma _{j}^{n} =\sigma (D_{j}^{n})$ is defined only where the discrete gradient $D_{j}^{n}$ is not vanishing. When the gradient is zero the MCM is consistent with the heat equation, see [14, 22]. Then, in the case $\vert D_{j}^{n}\vert \leq C\Delta {x}^{s}$, the numerical solution is obtained as an average of the neighboring values, so that on these points the scheme results to be consistent with the following heat equation

$$u_{t} = \frac{\Delta {x}^{2}} {4\Delta t} \Delta u.$$

We show an application of the scheme to filter noisy image. The noisy image is obtained adding Gaussian white noise of mean 0 and variance (ν ∕ 255)² to a given clean image, I _ex (note that the noise parameter has been denoted by ν to avoid confusions). In all the tests we have chosen a cubic interpolation, and in order to deal with non-smooth data we have regularized the gradient using G ∗ ∇ u, where G is a smoothing kernel. Moreover we have used the following approximation $(G {\ast}\nabla u)(x_{j},t_{n}) \simeq \frac{1} {4}\sum _{i\in \mathcal{D}(j)}D_{i}^{n}$. The errors are measured by a relative L ₁ discrete norm $\|\cdot \|_{1,r}$, defined as follows:

$$\|I_{\mathit{ex}}(\cdot ) - u_{\cdot }^{N}\|_{ 1,r} := \frac{\displaystyle\sum _{x_{j}\in \mathcal{G}_{\Delta x}}\vert I_{\mathit{ex}}(x_{j}) - u_{j}^{N}\vert } {\displaystyle\sum _{x_{j}\in \mathcal{G}_{\Delta x}}\vert I_{\mathit{ex}}(x_{j})\vert }.$$

In Figs. 9.6 and 9.7 we show the clean image, the noisy image and the restored image.

Fig. 9.6

Original, noisy (ν = 12) and filtered image by MCM (C = 0. 02, s = 1, Δx = 1, Δt = 0. 25, N = 5)

Fig. 9.7

Original, noisy (ν = 25) and filtered image by MCM (C = 0. 02, s = 1, Δx = 1, Δt = 0. 25, N = 5)

In Fig. 9.8 we show the relative L ₁ discrete errors evolution with respect to t. We see that in few iterations we reached the minimization of the L ₁ errors.

Fig. 9.8

L ₁ errors (y − axis) and time iteration (x − axis) for MCM model applied to a noisy images with ν = 12 (left) and ν = 25 (right)

To describe the SL scheme for the AMSS model, we first need to remind that the equation (9.45) in this case can be rewritten as following:

$$u_{t} ={ \left (\widehat{\sigma }{(\nabla u)}^{\top }{D}^{2}u\widehat{\sigma }(\nabla u)\right )}^{1/3},$$

where $\widehat{\sigma }(\nabla u) = {(\nabla u)}^{\perp }$, see for instance [34]. Then we introduce an extra parameter ρ, we define $\widehat{\sigma }_{j}^{n} :=\widehat{\sigma } (D_{j}^{n})$ and we take a directional second finite difference:

$$\widehat{\sigma }{(\nabla u)}^{\top }{D}^{2}u\widehat{\sigma }(\nabla u)(x_{ j},t_{n}) \simeq { \frac{u(x_{j} +\rho \widehat{\sigma }_{ j}^{n},t_{n}) + u(x_{j} -\rho \widehat{\sigma }_{j}^{n},t_{n}) - 2u_{j}^{n}} {\rho }^{2}}.$$

(9.47)

Finally, the SL scheme is written as

$$u_{j}^{n+1} \equiv \left \{\begin{array}{@{}l@{\quad }l@{}} w_{j}^{n+1} \quad &\mbox{ if }\vert D_{j}^{n}\vert > C\Delta x \\ \frac{1} {4}\mathop{\displaystyle\sum }\limits_{i \in \mathcal{D}(j)}w_{i}^{n+1}\quad &\mbox{ if }\vert D_{ j}^{n}\vert \leq C\Delta x \end{array} \right.$$

where

$$w_{i}^{n+1} \equiv \left \{\begin{array}{@{}l@{\quad }l@{}} u_{i}^{n}\,+\,\Delta t{\left (\dfrac{\Pi [{u}^{n}](x_{i} +\rho \widehat{\sigma }_{ i}^{n}) + \Pi [{u}^{n}](x_{i} -\rho \widehat{\sigma }_{i}^{n}) - 2u_{i}^{n}} {{2\rho }^{2}} \right )}^{1/3}\quad &\mbox{ if }\vert D_{i}^{n}\vert \,>\,C\Delta x \\ u_{i}^{n} \quad &\mbox{ if }\vert D_{ i}^{n}\vert \,\leq \,C\Delta x. \end{array} \right.$$

Such scheme as been proposed in [12], where a convergence result for its monotonic version is proved.

Let us notice that in this case the vector $\widehat{\sigma }$ is always defined, anyhow we need to consider differently the case $\widehat{\sigma }= 0$. In fact, if we would use the approximation (9.47) on the points where $\widehat{\sigma }= 0$, no diffusion would happen. On these points, we extend the solution by continuity: we take an average of the numerical solution, computed at the previous step when $\widehat{\sigma }$ is small, computed using (9.47) when $\widehat{\sigma }$ is big.

As for the previous model, we have chosen a cubic interpolation and regularized the gradient.

In Fig. 9.9 and in Fig. 9.10, we show respectively the clean image, the noisy image and the restored image. In Fig. 9.11 we show the relative L ₁ discrete errors evolution with respect to t. We see that in few iterations the minimum error is reached.

Fig. 9.9

Original, noisy (ν = 12) and filtered image by AMSS (C = 0. 02, s = 1, Δx = 1, Δt = 0. 2, $\rho = {(\Delta t)}^{1/6}$, N = 10)

Fig. 9.10

Original, noisy (ν = 25) and filtered image by AMSS (C = 0. 02, s = 1, Δx = 1, Δt = 0. 25, $\rho = {(\Delta t)}^{1/6}$, N = 10)

Fig. 9.11

Errors for MCM model for noisy images (leftν = 1, rightν = 25)

5 Segmentation via the LS Method

The segmentation problem in computer vision, as formulated by Mumford and Shah [40], can be defined in the following way: given an observed image I ₀ : Ω → [0, 1], find a decomposition of disjoint open sets $\Omega =\bigcup _{i}\overline{\Omega _{i}}$ and C = ⋃ _i ∂Ω _i , such that I ₀ varies smoothly within each Ω _i and rapidly or discontinuously across the boundaries of Ω _i . We denote by | C | the length of curves which belong to C.

A classical way to solve this problem is solving the following minimization problem:

$$\inf \limits _{I,C}{F}^{\mathit{MS}}(I,C)$$

(9.48)

where

$$\displaystyle\begin{array}{rcl}{ F}^{\mathit{MS}}(I,C) =\lambda \displaystyle\int _{ \Omega }{(I - I_{0})}^{2}\mathit{dx} +\nu \displaystyle\int _{ \Omega \setminus C}\vert \nabla I{\vert }^{2}\mathit{dx} +\mu \vert C\vert & &\end{array}$$

(9.49)

and μ, ν are fixed parameters of weight for the different terms of the functional. For a minimum $({I}^{{\ast}},{C}^{{\ast}})$, I ^∗ is an optimal piecewise smooth approximation of the initial image I ₀, and C ^∗ approximates the edges of I ₀. I ^∗ will be smooth on Ω ∖ C and will have its discontinuities on C ^∗.

Theoretical results of existence of minimizers for this problem can be found in Mumford and Shah [40], Morel and Solimini [38] and De Giorgi et al. [26].

A reduced case can be obtained by restricting the segmentation image I to piecewise constant functions, i.e. for $c_{i} = \mathit{mean}(I_{0})$ in Ω _i then I ≡ c _i inside each Ω _i . In this event, for the special case $i = \left \{0,1\right \}$ (that is the classical case of segmenting one object from the background image) the problem (9.48) becomes minimizing

$${E}^{\mathit{MS}}(c_{ 0},c_{1},C) =\lambda \displaystyle\int _{\Omega _{0}}{(I_{0} - c_{0})}^{2}\mathit{dx} +\lambda \displaystyle\int _{ \Omega _{1}}{(I_{0} - c_{1})}^{2}\mathit{dx} +\mu \vert C\vert.$$

(9.50)

We want now to use LS methods to solve this problem. LS methods are a very useful tool for computing evolving contours since they accommodate topological changes and allow to compute on a simple structured grid. For LS methods the curve C is defined as the zero-level set of a sufficiently regular function ϕ called level set function, i.e. ϕ(x) < 0 for x ∈ Ω ₀ and ϕ(x) > 0 for x ∈ Ω ₁.

In this formulation, denoting by H(ϕ) the usual Heaviside function, (9.50) becomes (see [17])

$$\displaystyle\begin{array}{rcl}{ E}^{V C}(c_{ 0},c_{1},\phi )& =& \lambda \displaystyle\int _{\Omega }{(I_{0} - c_{0})}^{2}H(\phi )\mathit{dx} +\lambda \displaystyle\int _{ \Omega }{(I_{0} - c_{1})}^{2}(1 - H(\phi ))\mathit{dx} \\ & & +\mu \displaystyle\int _{\Omega }\vert \nabla H(\phi )\vert \mathit{dx}. \end{array}$$

(9.51)

Considering H _ε and δ _ε two C ¹ regular approximations of the Heaviside function H and of the delta function so that $H_{\epsilon }^\prime =\delta _{\epsilon }$, we can write the Euler-Lagrange equation corresponding to the energy E ^VC obtaining

$$\displaystyle\begin{array}{rcl} \frac{\partial \phi } {\partial t}& =& \delta _{\epsilon }(\phi )\left [\mu \mbox{ div}\left ( \frac{\nabla \phi } {\vert \nabla \phi \vert }\right ) -\lambda {(I_{0} - c_{0})}^{2} +\lambda {(I_{ 0} - c_{1})}^{2}\right ] \\ \mbox{ where }\quad c_{0}& =& \frac{\displaystyle\int _{\Omega }I(x)H(\phi )\mathit{dx}} {\displaystyle\int _{\Omega }H(\phi )\mathit{dx}} ,\quad c_{1} = \frac{\displaystyle\int _{\Omega }I(x)(1 - H(\phi ))\mathit{dx}} {\displaystyle\int _{\Omega }(1 - H(\phi ))\mathit{dx}}.\qquad \end{array}$$

(9.52)

This is a time dependent equation converging to a steady solution witch is a minimizer of the functional (9.50).

A standard rescaling can be made, as in Zhao et al. [49], by replacing δ(ϕ) by | ∇ ϕ | . This time rescaling does not affect the steady state solution, but helps removing stiffness near the zero level sets of ϕ. Finally, we get the following nonlinear evolution equation:

$$\frac{\partial \phi } {\partial t} = \vert \nabla \phi \vert \left [\mu \,\mathrm{div}\left ( \frac{\nabla \phi } {\vert \nabla \phi \vert }\right ) -\lambda {(I_{0} - c_{0})}^{2} +\lambda {(I_{ 0} - c_{1})}^{2}\right ]$$

(9.53)

which is solved for t tending to + ∞, yielding the solution of the segmentation problem.

5.1 SL Scheme for Segmentation via the LS Method

In this section, we describe a SL approximation for Eq. (9.53). This equation has a first-order and a second-order term, which we approximate using SL schemes introduced in previous sections. We call c(x) the speed of propagation of the first-order term,

$$c(x) = -\lambda {(I_{0} - c_{0})}^{2} +\lambda {(I_{ 0} - c_{1})}^{2},$$

(9.54)

and using (9.20) we obtain

$$\frac{\partial \phi } {\partial t} =\mu \,\mathrm{ div}\left ( \frac{\nabla \phi } {\vert \nabla \phi \vert }\right )\vert \nabla \phi \vert + c(x)\max _{a\in B(0,1)}\left \{a \cdot \nabla \phi \right \}.$$

(9.55)

As in every SL approximation, we want to build first an approximation of the directional derivative. This implies that the velocity c(x) will appear directly inside the argument of the unknown function ϕ. For this particular problem, we have to take care of the change of sign of the velocity since will produce a sign change also in our scheme. We can rewrite (9.55) as

$$\begin{array}{l} \frac{\partial \phi } {\partial t} = \left \{\begin{array}{ll} \mu \,\mathrm{div}\left ( \dfrac{\nabla \phi } {\vert \nabla \phi \vert }\right )\vert \nabla \phi \vert +\max \limits _{a\in B(0,1)}\left \{c(x)a \cdot \nabla \phi \right \}, &\mbox{ where }c(x) \geq 0 \\ \mu \,\mathrm{div}\left ( \dfrac{\nabla \phi } {\vert \nabla \phi \vert }\right )\vert \nabla \phi \vert -\max \limits _{a\in B(0,1)}\left \{-c(x)a \cdot \nabla \phi \right \},&\mbox{ where }c(x) < 0. \end{array} \right. \end{array}$$

(9.56)

Next, we define $\phi _{j}^{n} =\phi (x_{j},t_{n})$ where $(x_{j},t_{n}) \in \mathcal{G}_{\Delta x,\Delta t}$.

Using, the schemes (9.46) for the second order term, and (9.22) for the first order term, and explicit Euler finite differences for the temporal derivative, Eq. (9.56) becomes

$$\displaystyle\begin{array}{rcl} \frac{\phi _{j}^{n+1} -\phi _{j}^{n}} {\Delta t} & =& \mu \frac{\left \{\dfrac{1} {2}\Pi \left [{\phi }^{n}\right ](x_{j} +\sigma _{ j}^{n}\sqrt{\Delta t}) + \dfrac{1} {2}\Pi \left [{\phi }^{n}\right ](x_{j} -\sigma _{j}^{n}\sqrt{\Delta t})\right \} -\phi _{j}^{n}} {\Delta t} \\ & & \quad -\mathit{sign}\left (c(x_{j})\right )\min _{a\in B(0,1)}\frac{\Pi [{\phi }^{n}](x_{j} + ac(x_{j})\Delta t) -\phi _{j}^{n}} {-\Delta t} ,\end{array}$$

(9.57)

where we have used the fact that $-\max (-p) =\min (p)$ and $\min _{a\in B(0,1)}(a \cdot f) =\min _{a\in B(0,1)}(-a \cdot f)$ to simplify the notations.

Then, we get the fully discrete SL scheme:

$$\displaystyle\begin{array}{rcl} \phi _{j}^{n+1}& =& \phi _{ j}^{n} +\mu \left \{\frac{1} {2}\Pi \left [{\phi }^{n}\right ](x_{ j} +\sigma _{ j}^{n}\sqrt{\Delta t}) + \frac{1} {2}\Pi \left [{\phi }^{n}\right ](x_{ j} -\sigma _{j}^{n}\sqrt{\Delta t}) -\phi _{ j}^{n}\right \} \\ & & +\mathit{sign}\left (c(x_{j})\right )\left \{\min _{a\in B(0,1)}\left \{\Pi [{\phi }^{n}](x_{ j} + ac(x_{j})\Delta t)\right \} -\phi _{j}^{n}\right \}, \end{array}$$

(9.58)

and coherently with (9.46) in the case of discrete space gradient $\vert D_{j}^{n}\vert \leq C\Delta {x}^{s}$.

We used this scheme to segment a variety of images, the results are shown in Figs. 9.12–9.14.

Fig. 9.12

Segmentation: μ = 0. 1, λ = 10⁴, two iterations

Fig. 9.13

Segmentation: μ = 1, λ = 0. 3 ∗ 10⁴, two iterations

Fig. 9.14

Segmentation: μ = 1, λ = 0. 3 ∗ 10⁴, two iterations

The most interesting feature of this approach is that the method is very rapid. While in classical finite difference schemes the information runs just one pixel for iteration, in this case we take information from a larger set. In particular to compute the value on the grid point x _j we use all the values on the grid nodes that are contained in a ball centered in x _j with radius c(x _j ). This is the reason of the rapid convergence of the scheme. In our tests we found the correct segmentation of the image always after 3 or less iterations. However, this is an experimental observation, a detailed treatment of SL scheme performances on this problem is still missing.

6 The Motion Segmentation Problem

Optical flow is the motion field in an image sequence; this problem belongs to the more general problem of motion estimation. Motion estimation and segmentation are strongly related topics that can increase the performances from each other. In particular segmentation can help us to solve the classical ambiguity near motion boundaries.

Both topics are traditional in computer vision. Various approaches to optical flow computation have been suggested in [5], especially variational techniques based on modifications of the method of Horn and Schunck [36] have yielded very convincing results.

Also in segmentation, variational techniques perform well. As we showed in the previous section, segmentation can be obtained minimizing the functional (9.48). In recent years the functional optimization is computed using LS methods [43].

In [3] and in [10], the authors propose to compute optical flow coupling estimators for optical flow and segmentation problem in a Mumford and Shah-like functional. Due to the presence of unspecified discontinuities in the integration domain, the minimization of this kind of functionals is difficult and we are going to solve it using level set techniques.

We will use the studied semi-Lagrangian schemes for the Mean Curvature Motion in this situation, to compute the front propagation of a level set function. Later, we compare the results for this application, in terms of accuracy and speed. The variational model which we use, is based on the optical flow functional used in various works (for a short summary the dedicated section in [9]) and the segmentation model presented in [16].

Given two images $I_{0}(x),I_{1}(x)\,:\,\Omega \,\rightarrow \,\mathbb{R}$, we have to find, at each point x ∈ Ω the optical flow vector $w(x) : {\mathbb{R}}^{2}\,\rightarrow \,{\mathbb{R}}^{2}$ that describes the shift of the pixel at x in the image I ₀ to the new location x + w in the image I ₁. Let us assume the following classical constraints: $I_{0}(x + w)\,=\,I_{1}(x)$ this is grey level constancy. It means that a pixel does not change its brightness from an image to the next. The second important principle is gradient constancy, an assumption that makes the model robust with respect to global additive illumination changes in the scene. So, our term of consistency with the data is

$$f_{\mathit{of }}(w) = k\vert I_{0}(x + w) - I_{1}(x){\vert }^{2} +\gamma \vert \nabla I_{ 0}(x + w) -\nabla I_{1}(x){\vert }^{2}.$$

(9.59)

The minimization of an optical flow functional with only the image constancy assumptions is ill-posed, to overcome this problem, a smoothness term regularizing the flow field is considered. The typical smoothness term as in [36] is

$$s_{\mathit{of }}(w) = \vert \nabla w{\vert }^{2}.$$

(9.60)

Using this term, nevertheless, we introduce regularization of the field and the known ambiguities along the discontinuities. For this reason, we place the smoothness term in a Mumford-Shah-like functional that provide to add regularization only in the regions of regularity of the solution.

Finally, calling Γ the discontinuity set of the optical flow, and | Γ | the length of the curve, we want to minimize the following functional

$$\displaystyle\begin{array}{rcl} E(w,\Gamma )& =& \displaystyle\int _{\Omega }k\vert I_{0}(x + w) - I_{1}(x){\vert }^{2} +\gamma \vert \nabla I_{ 0}(x + w) -\nabla I_{1}(x){\vert }^{2}\mathit{dx} \\ & & +\mu \displaystyle\int _{\Omega \setminus \Gamma }\vert \nabla w{\vert }^{2}\mathit{dx} +\nu \vert \Gamma \vert. \end{array}$$

(9.61)

This energy functional follows the principle of motion competition proposed for example in [23] and in [3], that was inspired by the work on segmentation by Vese and Chan [48]. This energy drives the interface of discontinuity and, simultaneously, makes an estimation of the optical flow in the different regions of smoothness.

In the optimization problem related to (9.61), if at the same time the optical flow, the number or regions separated by Γ and the position of the curve Γ are all unknown, then minimization can be hard. For this reason, in the next section we will consider the following simpler case. The motion field has an interface of discontinuity that divides the domain Ω in only two regions; inside these regions the motion field is constant. This choice is made for simplicity, and because, in this section, our aim is to present an easly understandable case. However, more general situations can be considered, e.g. in [10].

6.1 SL Scheme for the Motion Segmentation Problem

We want to deal with the functional (9.61) in a similar way as in segmentation, so we define a level set function ϕ such that the interface of discontinuity of the motion field Γ is the 0-level set of ϕ. Then, using the Euler-Lagrange equation and considering a time dependent equation converging to a steady solution which is a minimizer of the energy functional (9.61), we get the following evolutive equation:

$$\displaystyle\begin{array}{rcl} \frac{\partial \phi } {\partial t} = \vert \nabla \phi \vert \left [\nu \,\mathrm{div}\left ( \frac{\nabla \phi } {\vert \nabla \phi \vert }\right ) + [f_{\mathit{of }}({w}^{+}) - f_{\mathit{ of }}({w}^{-})] -\mu \left [s_{\mathit{ of }}({w}^{+}) - s_{\mathit{ of }}({w}^{-})\right ]\right ]& &\end{array}$$

(9.62)

where, w ⁺ and w ⁻ are the constant values of w inside and outside Γ: we discretize this equation with the same technique as in (9.58). We obtain the following numerical scheme:

$$\displaystyle\begin{array}{rcl} \phi _{j}^{n+1}& =& \phi _{ j}^{n} +\nu \left \{\frac{1} {2}\Pi \left [{\phi }^{n}\right ](x_{ j} +\sigma _{ j}^{n}\sqrt{\Delta t}) + \frac{1} {2}\Pi \left [{\phi }^{n}\right ](x_{ j} -\sigma _{j}^{n}\sqrt{\Delta t}) -\phi _{ j}^{n}\right \} \\ & & +\mathit{sign}(c(x_{j}))\left \{\min _{a\in B(0,1)}\left \{\Pi [{\phi }^{n}](x_{ j} + ac(x_{j})\Delta t)\right \} -\phi _{j}^{n}\right \}, \end{array}$$

(9.63)

where the first-order velocity term c(x _j ) is

$$c(x_{j}) = \left [f_{\mathit{of }}(w_{j}^{+}) - f_{\mathit{ of }}(w_{j}^{-})\right ] -\mu \left [s_{\mathit{ of }}(w_{j}^{+}) - s_{\mathit{ of }}(w_{j}^{-})\right ].$$

(9.64)

We used this scheme for two simple tests obtaining the results shown in Fig. 9.15 and 9.16. The first is a synthetic sequence where a ball is moving on a background, the second one is a small box taken from a more complicated real sequence (available on the site [47]). In these tests we have small displacements so we do not need a coarse-to-fine framework to capture larger ones.

Fig. 9.15

Motion segmentation: μ = 0. 4, k = 0. 5 ∗ 10⁵, ν = 2, γ = 0. 2, convergence in five iterations

Fig. 9.16

Motion segmentation: μ = 0. 4, κ = 0. 5 ∗ 10⁵, ν = 2, γ = 0. 2, convergence in four iterations

7 Conclusions

As we have seen in the previous sections, the approximation schemes based on the semi-Lagrangian approximation of first and second order partial differential equations can be successfully applied to the solution of classical image processing problems. The main features of SL schemes are the following:

• – They mimic the method of characteristics and give even in their simplest version a more accurate approximation with respect to first order finite difference schemes;

• – High-order approximation schemes can be obtained coupling high-order methods for integrating back along the characteristics (e.g. Heun, Runge-Kutta) with high-order interpolation techniques (e.g. polynomial, essentially non-oscillatory (ENO), weighted ENO);

• – They can be applied to structured and unstructured grids, so they are well suited also to problems where there is a need to use triangular grids as, for example, in computer vision;

• – SL approximation allows to obtain accurate schemes even for second order nonlinear degenerate equations. Such approximations are fully explicit (i.e. with moderate computing cost) and allow for large time steps. It is important to note that the last property is crucial when we look to the asymptotic behavior for t tending to + ∞. Since this implies a time integration over a large interval. a large time step guarantees a fast convergence to the steady state;

• – SL schemes have shown to be rather robust when applied to non smooth data, as noisy images. Moreover, in the time marching evolution there is less need for a re-initialization in order to keep the level sets and acceleration techniques (such as Fast Marching methods, [24, 45]) can be implemented.

A detailed analysis of these methods for first order model problems can be found in the book [32] and some of the results for second order problems can be found in [12, 14]. We conclude observing that there are still many open problems in the theory of SL schemes, e.g. in the construction and analysis of high-order approximations and acceleration methods (fast sweeping, fast marching). We are confident that these methods will be also useful for new challenging applications in image processing.