1 Introduction

The segmentation with depth information is a 2.1D sketch problem [1,2,3] in computer vision. Its goal is to reconstruct the complete shapes of occluded objects and their ordering relation in a specific scene based on only one single image via using segmentation techniques. Such segmentation plays an important role in image analysis and computer vision, such as object recognition and tracking. It is also a fundamental preprocessing step to some more complicated problems such as illusory shape recovery [4, 5].

In early 1990s, Nitzberg et al. [6] innovatively modeled the segmentation with depth as a minimization problem for an energy functional, which is called the NMS model. This model combines the classical Mumford–Shah model [7] for variational image segmentation and Euler’s elastica terms [8]. In order to optimize the NMS model in a tractable manner, authors in [6] have decomposed such problem into three successive steps: finding the edges and T-junctions in the image; hypothesizing the ordering relation of objects to be segmented and obtaining the associated minimum of energy functional; determining the objects’ ordering and the reconstructed shapes according to the minimum functional in the second step. This NMS model is a foundation for this type of segmentation problems associated with depth information, and many possible solutions have been proposed based on this model as described below.

Instead of the tedious multiple step procedure aforementioned, Esedoglu and March [9] transformed the original NMS model into an equivalent one based on \(\Gamma \)-convergence concept [10, 11] and its aim is to approximate the terms associated with length and elastica. Such derived model was claimed and demonstrated to be more tractable without requirement of T-junctions detection. For the computational efficiency, they designed a fast scheme by combing the semi-implicit discretization and fast Fourier transform (FFT) methods. But due to the over simplification of curvature-related terms in quadratic form, their model cannot preserve the corners of the recovered shapes, as they pointed out in [9].

Another rigorous approximation for the NMS model is based on variational level set method [13,14,15,16], which was proposed by Zhu et al. [12]. By making use of the standard variational method, they have derived a series of higher-order evolution equations including level set functions and then proposed the Smereka’s semi-implicit method [17] with FFT solver. However, it is difficult to discretize the fourth-order nonlinear partial differential equations (PDEs) in their approach. Additionally, they did not consider the definition of level set functions as signed distance functions during computation process as did in [12].

Recently, another similar work was conducted by Zhu et al. [18], where they proposed the Chan–Vese–Euler model by combining the Chan–Vese model and Euler elastica regularizer with an aim to recover the missed parts of contours. In order to minimize the proposed model efficiently, they designed a fast alternating direction method of multipliers (ADMM) [19, 20]. Although this model can integrate the missing parts to form a complete meaningful object, it was designed only for one foreground shape recovery problem without considering segmentation with depth information. Moreover, their work was based on binary label function method [22] or piecewise constant level set function method [21].

The curvature-related terms play crucial roles in the NMS model, but they also brought computational complexity due to the nonlinear higher-order derivatives. This issue also appears in other variational models of imaging sciences on the non-texture image inpainting [23, 24], illusory contours reconstruction [4, 5], image denoising [25,26,27,28] with feature (edge, corner, smoothness, contrast, etc.) preserving. In order to improve the computational efficiency and avoid solving nonlinear higher-order PDEs, some fast ADMM methods are systematically investigated in [27,28,29,30] for energy minimization problems associated with curvature terms.

Motivated by works in [27,28,29,30], in this paper, we focus on developing a fast ADMM method for optimizing the NMS model [6], which can be rewritten as a variational level set formulation [12]. The main difference between the variational level set formulation and the models in [23,24,25,26,27,28,29,30] is that the variational level set formulation in this paper is constrained by some nonlinear Eikonal equations of level set functions based on definition of signed distance functions. Usually, these constrained equations are guaranteed to be satisfied by solving a dynamic Hamilton–Jacobi equation using the upwind finite difference scheme [31], fast sweeping method [32], or they can be coped with penalty function method [33, 34] to get rid of these constraints.

The ADMM-projection (ADMM-P is used for abbreviation in the following parts of this paper) method proposed in this paper is based on the simplified variational level set formulation by replacing curvatures with Laplacians of level set functions on the premise that the Eikonal equations are satisfied during computation process. Another salient feature of this method is that the Eikonal equations can be satisfied indirectly by means of introducing auxiliary variables and implementing a direct projection [35, 36], which are suitable for the simplified variational level set formulation.

The paper is organized as follows: in Sect. 2, the variational level set formulation of the NMS functional along with its traditional gradient descent method (GDM is used for abbreviation in the follows) is reviewed; in Sect. 3, we derive the simplified variational level set formulation as an equivalent one of the NMS, and then design a fast ADMM-P method for the simplified variational level set formulation by transforming it into some sub-minimization problems for efficient computation in Sect. 4; in Sect. 5, extensive numerical experiments on different type images are conducted to illustrate the efficiency of the proposed model and its ADMM-P method. Concluding remarks are presented in Sect. 6.

2 The Variational Level Set Formulation for the NMS Model and its GDM Method

In this section, we will review the variational level set formulation of the NMS model first and then highlight our contributions clearly in Sects. 3 and 4. In [6], Nitzberg et al. defined the problem of segmentation with depth information as a problem of recovering occluded shapes and their ordering relations based on a 2D image. This problem is investigated with the following three assumptions: (1) there is no self-occlusion on the surface of each object; (2) the objects are not entangled with each other; (3) the pixel intensities of each object are approximately constant and different from each other. The variables defined in this problem are in three folds: the shapes of the regions \(R_1 ,R_2 ,\ldots ,R_n \) to which different objects belong; the ordering relations among objects; the pixel intensities of objects.

Without loss of generality, one can assume that the objects \(R_1 ,R_2 ,\ldots ,R_n \) in an image are in ascending order, i. e., \(R_1 \) is the nearest object to the observer while \(R_n \) is the farthest one (i. e. background). Let \(R^{\prime }_i \) be defined as the visible part of \(R_i \), i.e., \(R^{\prime }_1 =R_1 ,R^{\prime }_i =R_i -\bigcup \limits _{j<i} {R_j } ,\left( {i=2,\ldots ,n} \right) \). In addition, \(R^{\prime }_{n+1} =\Omega -\bigcup \limits _{j<n+1} {R_j } \) is defined as the visible background. Based on the above assumptions and definitions, the NMS functional is formulated as in [6]

$$\begin{aligned} E= & {} \sum _{i=1}^n {\int _{\partial R_i \cap \Omega } {\left[ {\alpha +\beta \phi \left( {\kappa _i } \right) } \right] } } \mathrm{d}s\nonumber \\&+\,\sum _{i=1}^{n+1} {\int _{R^{\prime }_i } {\left( {f(x)-c_i } \right) ^{2}} } \mathrm{d}x. \end{aligned}$$
(1)

where \(\alpha ,\beta \) are two positive penalty parameters, s is arc length, and \(c_i \in R_i \) are pixel intensities of the ith object, \(\kappa \) denotes the curvature of boundary for region \(R_i \). While one of the main difficulties in minimizing the above NMS functional is to calculate the energy term.

By making use of variational method [13,14,15] and level set method [16], Zhu and Chan proposed the variational level set formulation in [12] to replace the original NMS model. For such purpose, a level set function representing an interface \(\Gamma (t)\) (t represents time) is defined implicitly as the zero level set of a Lipchitz continuous function \(\varphi :R^{2}\rightarrow R\), which in turn can be defined in terms of time t as the following signed distance function for efficient computation,

$$\begin{aligned} \left\{ {\begin{array}{l} \varphi ({x,t})=d\left( {\Gamma (t),x} \right) ,\hbox {if}\,x\,\hbox {is}\,\hbox {inside}\, \Gamma (t)\\ \varphi ({x,t})=0,\hbox {if}\,x\,\hbox {is}\,\hbox {at}\,\Gamma (t)\\ \varphi ({x,t})=-d\left( {\Gamma (t),x} \right) ,\hbox {if}\,x\,\hbox {is}\,\hbox {outside}\,\Gamma (t) \\ \end{array}} \right. . \end{aligned}$$
(2)

where \(d({\Gamma (t),x})\) denotes the shortest Euclidean distance from x to \(\Gamma (t)\), and from which its Eikonal equation is derived as \(\left| {\nabla \varphi ({x,t})} \right| =1\).

For an image defined in domain \(\Omega \) with boundary \(\Gamma (t)\), according to the co-area and area formulas in [31], we will have

$$\begin{aligned} length(\Gamma )= & {} \int _\Omega {\left| {\nabla H(\varphi )} \right| \mathrm{d}x} \nonumber \\= & {} \int _\Omega {\delta (\varphi )\left| {\nabla \varphi } \right| } \mathrm{d}x, \end{aligned}$$
(3)
$$\begin{aligned}&{\textit{area}}(\Omega )=\int _\Omega {H(\varphi )\mathrm{d}x} . \end{aligned}$$
(4)

where \(H({\varphi (x)})\) and \(\delta ({\varphi (x)})\) are Heaviside function and Dirac delta function, respectively, with the definitions given below.

$$\begin{aligned} H({\varphi (x)})= & {} \left\{ {\begin{array}{ll} 1,&{}{\hbox {if}}\,\, \,\,\varphi (x)\ge \hbox {0} \\ 0,&{}{\hbox {otherwise}}\\ \end{array}} \right. , \end{aligned}$$
(5)
$$\begin{aligned} \delta ( {\varphi (x)} )= & {} \frac{\partial H( {\varphi (x)} )}{\partial \varphi (x)}. \end{aligned}$$
(6)

In fact, these kinds of original definitions are difficult to calculate as there exist non-differential functions. In practice, these two functions are often approximated via introducing a small positive parameter \(\varepsilon \) in (5) and (6) [10, 11] by denoting as \(H_\varepsilon ({\varphi (x)})\) and \(\delta _\varepsilon ({\varphi (x)})\). In this way, we can obtain some feasible computing forms of Heaviside function and Dirac delta function in [13,14,15,16], given by

$$\begin{aligned} H_\varepsilon (\varphi )= & {} \frac{\hbox {1}}{\hbox {2}}\left( {1+\frac{2}{\pi }\arctan \left( \frac{\varphi }{\varepsilon }\right) } \right) , \end{aligned}$$
(7)
$$\begin{aligned} \delta _\varepsilon (\varphi )= & {} \frac{1}{\pi }\frac{\varepsilon }{\varepsilon ^{2}+\varphi ^{2}}. \end{aligned}$$
(8)

By making use of these level set functions \(\varphi _i ( i=1,2,\ldots ,n )\), we can represent objects \(R_i \) as \(R_i =\{ x: H({\varphi _i (x)})=1 \}\), each visible part as \(R^{\prime }_i =R_i -\mathop {\bigcup }\limits _{j<i} R_j \) which is given by \(R_i^{\prime } =\{ {x:H({\varphi _i } )\prod \limits _{j=1}^{i-1} {( {1-H( {\varphi _j } )} )} =1} \}\), thus the term \(\sum \limits _{i=1}^{n+1} {\int _{R^{{\prime }}_i } {( {f(x)-c_i } )} ^{2}\mathrm{d}x} \) in (1) can be rewritten as

$$\begin{aligned}&\sum _{i=1}^n {\left\{ {\int _\Omega {\left( {f-c_i } \right) ^{2}H\left( {\varphi _i } \right) \prod _{j=1}^{i-1} {\left( {1-H({\varphi _j })} \right) } } \mathrm{d}x} \right\} }\nonumber \\&\quad +\,\int _\Omega {\left( {f-c_{n+1} } \right) ^{2}\prod _{j=1}^n {\left( {1-H({\varphi _j })} \right) } \mathrm{d}x} . \end{aligned}$$
(9)

And more importantly, the curvature \(\kappa _i \) can be represented as \(\kappa _i =\nabla \cdot ( {\frac{\nabla \varphi _i}{|{\nabla \varphi _i } |}} )\). Now we choose \(\phi (\kappa )=|\kappa |\) as the authors of [12] did with the reason that the NMS model (1) can preserve the object corners when \(|\kappa |\) becomes large. Thus, the NMS functional (1) can be reformulated with level set functions as below and our major task is to solve the level set functions \(\varphi _i \)

$$\begin{aligned} E= & {} \sum _{i=1}^n {\int _\Omega {[ {\alpha +\beta \phi ( {\kappa _i } )} ]} | {\nabla \varphi _i } |\delta ( {\varphi _i } )\mathrm{d}x} \nonumber \\&+\,\sum _{i=1}^n {\left\{ {\int _\Omega {( {f-c_i } )^{2}H( {\varphi _i } )\prod _{j=1}^{i-1} {( {1-H( {\varphi _j } )} )} } \mathrm{d}x} \right\} } \nonumber \\&+\,\int _\Omega {( {f-c_{n+1} } )^{2}\prod _{j=1}^n {( {1-H( {\varphi _j } )} )} \mathrm{d}x} \end{aligned}$$
(10)

Now, by using the standard variational method, one can obtain the following GDM equations of level set functions via minimization of functional (10).

$$\begin{aligned} \frac{\partial \varphi _i }{\partial t}= & {} \left| {\nabla \varphi _i } \right| \nabla \cdot \left[ \frac{\nabla \varphi _i }{\left| {\nabla \varphi _i } \right| }\Psi ({\kappa _i } )-\frac{1}{\left| {\nabla \varphi _i } \right| }\left( \nabla \left( {\phi ^{\prime }\left( {\kappa _i } \right) \left| {\nabla \varphi _i } \right| } \right) \right. \right. \nonumber \\&\left. \left. -\,\left( {\frac{\nabla \varphi _i }{\left| {\nabla \varphi _i } \right| }\cdot \nabla ( {\phi ^{\prime }({\kappa _i })\left| {\nabla \varphi _i } \right| })} \right) \frac{\nabla \varphi _i }{\left| {\nabla \varphi _i } \right| } \right) \right] \nonumber \\&-\,\left| {\nabla \varphi _i } \right| \left( {f-c_i } \right) ^{2}\prod _{j=1}^{i-1} {\left( {1-H({\varphi _j })} \right) } \nonumber \\&+\,\left| {\nabla \varphi _i } \right| \sum _{s=i+1}^{n+1} \left\{ \left( {f-c_s } \right) ^{2}H\left( {\varphi _s } \right) \prod _{j=1}^{i-1} \right. \nonumber \\&\left. {\left( {1-H( {\varphi _j } )} \right) \prod _{j=i+1}^{s-1} {\left( {1-H({\varphi _j })} \right) } } \right\} \end{aligned}$$
(11)

where \(i=1,2,\ldots ,n\) denotes the number of objects in the image, and \(H({\varphi _{n+1} })=1\) is introduced only for consistency of description. In fact, (11) is a standard Hamilton–Jacobi equation, which can be solved iteratively using some explicit upwind-alike schemes. However, the convergence of such iteration heavily depends on selected time step size. In order to overcome this problem and improve the computational stability, the authors in [12] employed the Smereka’s semi-implicit iteration method [17] to relax the Courant–Friedrichs–Lewy condition. In order to improve the computational efficiency further, they employed the FFT method to solve the discretized equations. But discretization of the nonlinear fourth-order derivatives is tedious and is prone to errors. More seriously, they did not pay attention to the requirement of level set functions as the signed distance functions during computational iteration. In order to overcome these computational problems, we will propose a new ADMM projection approach in this paper.

3 ADMM-P Method for Simplified Variational Level Set Formulation

Instead of solving (10) directly by using the conventional variational method and solving the related GDM equations as mentioned in previous section, in this section, we first transform the original minimization problem into several sub-optimization problems by introducing some auxiliary variables and then solve them using the alternating directional optimization strategy.

First, we propose a simplified model of the original problem by considering the property of level set functions as the signed distance functions. If the level set functions are treated as the signed distance functions during computational process with property \(| {\nabla \varphi _i } |=1(i=1, 2,{\ldots }n)\), the curvatures in (10) can be replaced by Laplacians of level set functions, and in this case \(\kappa _i =\nabla \cdot ( {\frac{\nabla \varphi _i }{| {\nabla \varphi _i } |}} )=\nabla \cdot ( {\nabla \varphi _i } )=\Delta \varphi _i \) can be reduced to \(\kappa _i =\nabla \cdot ( {\nabla \varphi _i } )\). Based on this observation, the NMS functional (10) can be rewritten as the following simplified version

$$\begin{aligned} E= & {} \sum _{i=1}^n {\int _\Omega {\left[ {\alpha +\beta \left| {\nabla \cdot \left( {\nabla \varphi _i } \right) } \right| } \right] } \delta \left( {\varphi _i } \right) \mathrm{d}x} \nonumber \\&+\,\sum _{i=1}^n {\left\{ {\int _\Omega {\left( {f-c_i } \right) ^{2}H\left( {\varphi _i } \right) \prod _{j=1}^{i-1} {\left( {1-H({\varphi _j })} \right) } } \mathrm{d}x} \right\} }\nonumber \\&+\,\int _\Omega {\left( {f-c_{n+1} } \right) ^{2}\prod _{j=1}^n {\left( {1-H({\varphi _j })} \right) } \mathrm{d}x} \end{aligned}$$
(12a)
$$\begin{aligned}&\hbox {s.t.} \left| {\nabla \varphi _i } \right| =1. \end{aligned}$$
(12b)

Note that Eqs. (12a) and (12b) are equivalent to the original problem (15). Therefore, some fast ADMM algorithms can be directly applied to (12a) and (12b). For such purpose, some auxiliary variables \(\vec {w}_i =[ {w_{i1} ,w_{i2} } ]^{T}\) and \(v_i \) with property \(\vec {w}_i \approx \nabla \varphi _i , v_i \approx \nabla \cdot \vec {w}_i \) and the Lagrangian multipliers \(\vec {\lambda }_{1i} ,\lambda _{2i} \) are introduced. In this case, the constraint \(| {\nabla \varphi _i } |=1\) will be replaced by \(| {\vec {w}_i } |=1\), and this substitution can avoid the traditional re-initialization process to satisfy \(| {\nabla \varphi _i } |=1\) by imposing the compulsory constraint for \(\vec {w}_i \). And \(| {\vec {w}_i } |=1\) then can be guaranteed by implementing a projection method easily. Based this observation, we can transform (12a) and (12b) into the following augmented Lagrangian functional with some explicit constraints.

$$\begin{aligned}&E({c,\varphi ,\vec {w},v,\vec {\lambda }_1 ,\lambda _2 } )\nonumber \\&\quad =\sum _{i=1}^n {\int _\Omega {\left[ {\alpha +\beta \left| {v_i } \right| } \right] } \delta \left( {\varphi _i } \right) \mathrm{d}x} +\,\sum _{i=1}^n {\int _\Omega {\vec {\lambda }_{1i} \cdot \left( {\vec {w}_i -\nabla \varphi _i } \right) } \mathrm{d}x} \nonumber \\&\quad \quad +\,\frac{\mu _1 }{2}\sum _{i=1}^n {\int _\Omega {\left( {\vec {w}_i -\nabla \varphi _i } \right) ^{2}} \mathrm{d}x} +\,\sum _{i=1}^n {\int _\Omega {\lambda _{2i} \left( {v_i -\nabla \cdot \vec {w}_i } \right) } \mathrm{d}x} \nonumber \\&\quad \quad +\,\frac{\mu _2 }{2}\sum _{i=1}^n {\int _\Omega {\left( {v_i -\nabla \cdot \vec {w}_i } \right) ^{2}} \mathrm{d}x,} \nonumber \\&\quad \quad +\,\sum _{i=1}^n \left\{ \int _\Omega {({f-c_i })}^{2}H( {\varphi _i } )\prod _{j=1}^{i-1} {\left( {1-H({\varphi _j })} \right) } \mathrm{d}x \right\} \nonumber \\&\quad \quad +\,\int _\Omega {\left( {f-c_{n+1} } \right) } ^{2}\prod _{j=1}^n {\left( {1-H({\varphi _j })} \right) } \mathrm{d}x \end{aligned}$$
(13a)
$$\begin{aligned}&\hbox {s.t.}\quad | {\vec {w}_i } |=1,\quad i=1,2,\ldots , n. \end{aligned}$$
(13b)

where \(\mu _1 ,\mu _2 \) are positive penalty parameters, and \(c=\left\{ {{\begin{array}{l} {c_1 }\quad {c_2 }\quad \ldots \quad {c_{n+1} } \\ \end{array} }} \right\} , \varphi =\left\{ {{\begin{array}{l} {\varphi _1 }\quad {\varphi _2 }\quad \ldots \quad {\varphi _n } \\ \end{array} }} \right\} ,\vec {w}=\left\{ {{\begin{array}{l} {\vec {w}_1 }\quad {\vec {w}_2 }\quad \ldots \quad {\vec {w}_n } \\ \end{array} }} \right\} , v=\left\{ {{\begin{array}{l} {v_1 }\quad {v_2 }\quad \ldots \quad {v_n } \\ \end{array} }} \right\} , \vec {\lambda }_1 =\left\{ {\vec {\lambda }_{11} \hbox { }\vec {\lambda }_{12} \ldots \vec {\lambda }_{1n} } \right\} , \lambda _2 =\left\{ {\lambda _{21} \hbox { }\lambda _{22} \ldots \lambda _{2n} } \right\} \). The approximations for \(\vec {w}_i \approx \nabla \varphi _i \) and \(v_i \approx \nabla \cdot \vec {w}_i \) can be achieved by the maximization with respect to \(\vec {\lambda }_{1i} \) and \(\lambda _{2i} \) in the energy functional (13.1). Meanwhile, the constraint \(| {\nabla \varphi _i } |=1\) is replaced by \(| {\vec {w}_i } |=1\), which can be guaranteed by the projection method directly. It is noteworthy that this constraint can also be guaranteed via penalty function method, in which larger computational expense would be inevitable. Next we will propose a new approach to solve this problem.

The proposed ADMM-P method can be implemented in finite steps with stopping criteria. In each step, we can calculate a sub-problem. Also, the sub-problem of minimization is carried out with respect to one variable while keeping other variables to be fixed temporarily. For the formulations (13a) and (13b), we first initialize the unknown values \(c^{0},\varphi ^{0},\vec {w}^{0},v^{0},\vec {\lambda }_1^0 ,\lambda _2^0 \), and then, we start a procedure of optimization step by step. In each step, the minimization of (13a) and (13b) can be divided into following sub-problems

$$\begin{aligned} c^{k+1}= & {} \hbox {Arg}\mathop {\min }\limits _c \left\{ {\varepsilon _0 (c)=E\left( {c,\varphi ^{k},\vec {w}^{k},v^{k},\vec {\lambda }_1^k ,\lambda _2^k } \right) } \right\} \end{aligned}$$
(14)
$$\begin{aligned} \varphi ^{k+1}= & {} \hbox {Arg}\mathop {\min }\limits _\varphi \left\{ {\varepsilon _1 (\varphi )=E\left( {c^{k+1},\varphi ,\vec {w}^{k},v^{k},\vec {\lambda }_1^k ,\lambda _2^k } \right) } \right\} \nonumber \\ \end{aligned}$$
(15)
$$\begin{aligned} \vec {w}^{k+1}= & {} \hbox {Arg}\mathop {\min }\limits _{\vec {w}} \left\{ {\varepsilon _1 \left( {\vec {w}} \right) =E\left( {c^{k+1},\varphi ^{k+1},\vec {w},v^{k},\vec {\lambda }_1^k ,\lambda _2^k } \right) } \right\} \nonumber \\\end{aligned}$$
(16a)
$$\begin{aligned}&\hbox {s.t.}\quad \left| {\vec {w}_i^{k+1} } \right| =1,i=1,2,\ldots , n \end{aligned}$$
(16b)
$$\begin{aligned} v^{k+1}= & {} \hbox {Arg}\mathop {\min }\limits _v \left\{ {\varepsilon _1 \left( v \right) =E\left( {c^{k+1},\varphi ^{k+1},\vec {w}^{k+1},v,\vec {\lambda }_1^k ,\lambda _2^k } \right) } \right\} \nonumber \\\end{aligned}$$
(17)
$$\begin{aligned} \vec {\lambda }_1^{k+1}= & {} \vec {\lambda }_1^k +\mu _1 \left( {\vec {w}^{k+1}-\nabla \varphi ^{k+1}} \right) \end{aligned}$$
(18)
$$\begin{aligned} \lambda _2^{k+1}= & {} \lambda _2^k +\mu _2 ( {v^{k+1}-\nabla \cdot \vec {w}^{k+1}} ) \end{aligned}$$
(19)

where (18) and (19) are responsible for updating of the Lagrange multipliers. These functionals of the above mentioned sub-problems are given, respectively, below.

$$\begin{aligned} \varepsilon _0 (c)= & {} \sum _{i=1}^n {\left\{ {\int _\Omega {\left( {f-c_i } \right) ^{2}H\left( {\varphi _i^k } \right) \prod _{j=1}^{i-1} {\left( {1-H\left( {\varphi _j^k } \right) } \right) } } \mathrm{d}x} \right\} } \nonumber \\&+\,\int _\Omega {\left( {f-c_{n+1} } \right) ^{2}\prod _{j=1}^n {\left( {1-H\left( {\varphi _j^k } \right) } \right) } \mathrm{d}x} \end{aligned}$$
(20)
$$\begin{aligned} \varepsilon _1 (\varphi )= & {} \sum _{i=1}^n {\int _\Omega {\left[ {\alpha +\beta \left| {v_i ^{k}} \right| } \right] } \delta \left( {\varphi _i} \right) \mathrm{d}x} \nonumber \\&+\,\sum _{i=1}^n {\int _\Omega {\vec {\lambda }_{1i}^k \cdot \left( {\vec {w}_i^k -\nabla \varphi _i} \right) } \mathrm{d}x} \nonumber \\&+\,\frac{\mu _1 }{2}\sum _{i=1}^n {\int _\Omega {\left( {\vec {w}_i^k -\nabla \varphi _i} \right) } ^{2}\mathrm{d}x} \nonumber \\&+\,\sum _{i=1}^n \left\{ \int _\Omega \left( {f-c_i^{k+1} } \right) ^{2}H({\varphi _{i}})\right. \nonumber \\&\left. \prod _{j=1}^{i-1} ({1-H({\varphi _{j}})}) \mathrm{d}x \right\} \nonumber \\&+\,\int _\Omega {\left( {f-c_{n+1}^{k+1} } \right) ^{2}\prod _{j=1}^n ( {1-H({\varphi _{j }})})} \mathrm{d}x \end{aligned}$$
(21)
$$\begin{aligned} \varepsilon _2 ({\vec {w}})= & {} \sum _{i=1}^n {\int _\Omega {\vec {\lambda }_{1i}^k \cdot \left( {\vec {w}_i -\nabla \varphi _i^{k+1} } \right) } \mathrm{d}x} \nonumber \\&+\,\frac{\mu _1 }{2}\sum _{i=1}^n {\int _\Omega {\left( {\vec {w}_i -\nabla \varphi _i^{k+1} } \right) }^{2}\mathrm{d}x}\nonumber \\&+\,\sum _{i=1}^n {\int _\Omega {\lambda _{2i}^k } } \left( {v_i^k -\nabla \cdot \vec {w}_i} \right) \mathrm{d}x\nonumber \\&+\,\frac{\mu _2 }{2}\sum _{i=1}^n {\int _\Omega {\left( {v_i^k -\nabla \cdot \vec {w}_i} \right) } ^{2}\mathrm{d}x} \end{aligned}$$
(22)
$$\begin{aligned} \varepsilon _3 ({\vec {v}} )= & {} \sum _{i=1}^n {\int _\Omega {[ {\alpha +\beta |{v_i}|} ]} \delta ( {\varphi _i^{k+1} } )\mathrm{d}x} \nonumber \\&+\,\sum _{i=1}^n {\int _\Omega {\lambda _{2i}^k } } ( {v_i -\nabla \cdot \vec {w}_i^{k+1} } )\mathrm{d}x\nonumber \\&+\,\frac{\mu _2 }{2}\sum _{i=1}^n {\int _\Omega {( {v_i -\nabla \cdot \vec {w}_i^{k+1} } )}^{2}\mathrm{d}x} \end{aligned}$$
(23)

In summary, we can present the ADMM-P approach in a pseudo-code format as follows.

figure g

Whether the above proposed ADMM method can be suitably applied for solving the non-convex and non-smooth problems in computer vision has attracted extensive attention. The authors in [39, 40] discussed this issue in detail and provided many applications in which many similar algorithms have been developed and successfully used to achieve excellent performances via solving a variety of non-convex problems. In addition, many other authors [18, 19, 26, 27, 29, 30] also have made a success of applying the ADMM method to different problems. In this paper, we adopt a similar research route to design a new algorithm to deal with the non-convex and non-smooth problem for the segmentation problem with depth.

Next we will consider each sub-problem in above algorithm individually.

4 Minimization of Each Sub-problem

4.1 Estimation of the Piecewise Constant Parameters

In the \(({k+1})\) step of the proposed ADMM, the average image intensity values \(c_i \) in different regions can be obtained by using the standard variational method based on (14) and (20), which are given by

$$\begin{aligned} c_i^{k+1}= & {} \frac{\int _\Omega {fH( {\varphi _i^k } )\prod \limits _{j=1}^{i-1} {( {1-H( {\varphi _j^k } )} )\mathrm{d}x} } }{\int _\Omega {H( {\varphi _i^k } )\prod \limits _{j=1}^{i-1} {( {1-H( {\varphi _j^k } )} )\mathrm{d}x} } },\nonumber \\&i=1,2,\ldots ,n, \end{aligned}$$
(24a)
$$\begin{aligned} c_{n+1}^{k+1}= & {} \frac{\int _\Omega {f\prod \limits _{j=1}^n {( {1-H( {\varphi _j^k } )} )} \mathrm{d}x} }{\int _\Omega {\prod \limits _{j=1}^n {( {1-H( {\varphi _j^k } )} )} \mathrm{d}x} }. \end{aligned}$$
(24b)

4.2 Calculation of the Level Set Functions

For the minimization sub-problems (15) and (21) with respect to the level set functions \(\varphi _i \), the corresponding Euler–Lagrange equations are given by

$$\begin{aligned} \left\{ {\begin{array}{l} [ {\alpha +\beta | {v_i^k } |} ]\delta ^{\prime }( {\varphi _i } )+\nabla \cdot \vec {\lambda }_{1i}^k +\mu _1 \nabla \cdot ( {\vec {w}_i^k -\nabla \varphi _i } ) \\ +\delta ( {\varphi _i } )( {f-c_i^{k+1} } )^{2}\prod _{j=1}^{i-1} {( {1-H( {\varphi _j } )} )} \\ -\delta ( {\varphi _i } )\left[ \sum _{s=i+1}^{n+1} \left\{ ( {f-c_s^{k+1} } )^{2}H( {\varphi _s } )\right. \right. \\ \left. \left. \prod _{j=1}^{i-1} ( {1-H( {\varphi _j } )} )\prod _{j=i+1}^{s-1} {( {1-H( {\varphi _j } )} )} \right\} \right] =0\hbox { in }\Omega \\ ( {\vec {\lambda }_{1i}^k +\mu _1 ( {\vec {w}_i^k -\nabla \varphi _i } )} )\cdot \vec {n}=0\quad \hbox { on }\partial \Omega \\ \end{array}} \right. ,\nonumber \\ \end{aligned}$$
(25a)

where \(\vec {w}_i^k ,\vec {v}_i^k \) are fixed temporarily when \(\varphi _i^{k+1}\) is calculated by using the semi-implicit difference scheme and Gauss–Seidel iterative method. To be more specific, the Gauss–Seidel iterative method can be used for the linear terms of (25a) \(\nabla \cdot \vec {\lambda }_{1i}^k +\mu _1 \nabla \cdot ( {\vec {w}_i^k -\nabla \varphi _i } )\), while the nonlinear terms:

$$\begin{aligned}&[ {\alpha +\beta | {v_i^k } |} ]\delta ^{\prime }( {\varphi _i } )+\delta ( {\varphi _i } )\cdot ( {f-c_i^{k+1} } )^{2}\\&\quad \prod _{j=1}^{i-1} {( {1-H({\varphi _j })} )} -\delta ( {\varphi _i } )\left[ \sum _{s=i+1}^{n+1} \left\{ ( {f-c_s^{k+1} } )^{2}H( {\varphi _s } )\right. \right. \\&\left. \left. \prod _{j=1}^{i-1} ( {1-H({\varphi _j })} )\prod _{j=i+1}^{s-1} {( {1-H({\varphi _j })} )} \right\} \right] \end{aligned}$$

need to be calculated by an explicit difference scheme. According to the standard technique introduced by Zhao and Chan [13], we use \(| {\nabla \varphi _i } |\) to replace \(\delta ( {\varphi _i } )\), which facilitates accelerating the evolution process [37], and then directly utilize \(|{\nabla \varphi _i }|=1\). The final calculation result of \(\varphi _i \) is shown as follows

$$\begin{aligned} \varphi _i^{k+1}= & {} \left\{ \left( \varphi _{i\left( {m-1,l} \right) }^{k+1} +\varphi _{i({m,l-1})}^{k+1} +\varphi _{i({m+1,l})}^k +\varphi _{i\left( {m,l+1} \right) }^k \right) \right. \nonumber \\&\left. -\,\nabla \cdot \vec {w}_i^k -{\nabla \cdot \vec {\lambda }_{1i}^k }/{\mu _1 } \right. \nonumber \\&\left. -\,\left( {f-c_i^{k+1} } \right) ^{2}\prod _{j=1}^{i-1} {\left( {1-H\left( {\varphi _j^k } \right) } \right) } /{\mu _1 } \right. \nonumber \\&\left. +\,\sum _{s=i+1}^{n+1} \left[ \left( {f-c_s^{k+1} } \right) ^{2}H({\varphi _s^k })\right. \right. \nonumber \\&\left. \left. \prod _{j=1}^{i-1} {( {1-H({\varphi _j^k })} )\prod _{j=i+1}^{s-1} {({1-H({\varphi _j^k } )} )} } \right] /{\mu _1 } \right. \nonumber \\&\left. -\,\left[ {\alpha +\beta |{v_i^k }|} \right] \delta ^{\prime }({\varphi _i^k })/{\mu _1 } \right\} /4, \end{aligned}$$
(25b)

where \(m=1,2,\ldots ,M, l=1,2,\ldots ,N\) and \(M\times N\) denotes the size of the image.

4.3 Minimization of the Auxiliary Variable \(\vec {w}_i \)

The Euler–Lagrange equations with respect to \(\vec {w}_i \) can be derived from (16) and (22) via calculus of variation when the variables \(\varphi _i^{k+1} ,\vec {v}_i^k \) are kept constant temporarily,

$$\begin{aligned} \left\{ \begin{array}{l} {\vec {\lambda }_{1i}^k} +\mu _1 \left( {\vec {w}_i -\nabla \varphi _i^{k+1} } \right) +\nabla \lambda _{2i}^k +\mu _2 \left( {v_i -\nabla \cdot \vec {w}_i } \right) =0 \\ \hbox {s.t.}\quad \left| {\vec {w}_i } \right| =1 \\ \end{array} \right. .\nonumber \\ \end{aligned}$$
(26)

The first equation of (26) can be solved by using the fast Fourier transform (FFT) [30]. We now detail the FFT implementation as follows

$$\begin{aligned}&\mu _1 \vec {w}_i -\mu _2 \nabla \left( {\nabla \cdot \vec {w}_i } \right) \nonumber \\&\quad =-\vec {\lambda }_{1i}^k -\nabla \lambda _{2i}^k +\,\mu _1 \nabla \varphi _i^{k+1} -\mu _2 \nabla v_i^k \end{aligned}$$
(27a)
$$\begin{aligned}&\left\{ {\begin{array}{l} \mu _1 w_{i1} -\mu _2 \left( {\partial _{xx} w_{i1} \hbox {+}\partial _{xy} w_{i2} } \right) \\ \quad =\mu _1 \partial _x \varphi _i^{k+1} -\lambda _{1i1}^k -\partial _x \lambda _{2i}^k -\mu _2 \partial _x v_i^k \\ \mu _1 w_{i2} -\mu _2 \left( {\partial _{yx} w_{i1} +\partial _{yy} w_{i2} } \right) \\ \quad =\mu _1 \partial _y \varphi _i^{k+1} -\lambda _{1i2}^k -\partial _y \lambda _{2i}^k -\mu _2 \partial _y v_i^k \\ \end{array}} \right. . \end{aligned}$$
(27b)

For clarity, we introduce \(g_{i1} =\mu _1 \partial _x \varphi _i^{k+1} -\lambda _{1i1}^k -\partial _x \lambda _{2i}^k -\mu _2 \partial _x v_i^k \) and \(g_{i2} =\mu _1 \partial _y \varphi _i^{k+1} -\lambda _{1i2}^k -\partial _y \lambda _{2i}^k -\mu _2 \partial _y v_i^k \) . Then we can rewrite (27b) as

$$\begin{aligned} \left\{ {\begin{array}{l} \mu _1 w_{i1} -\mu _2 \left( {\partial _{xx} w_{i1} {+}\partial _{xy} w_{i2} } \right) =g_{i1} \\ \mu _1 w_{i2} -\mu _2 \left( {\partial _{yx} w_{i1} +\partial _{yy} w_{i2} } \right) =g_{i2} \\ \end{array}} \right. . \end{aligned}$$
(28)

As in [27, 29, 30], after introducing the identity operator \(If( {m,l} )=f( {m,l} )\) and shifting operators \(S_x^\pm f( {m,l} )=f( {m\pm 1,l} ), S_y^\pm f( {m,l} )=f( {m,l\pm 1} )\), (28) can be rewritten as

$$\begin{aligned} \left\{ {\begin{array}{l} \mu _1 w_{i1} -\mu _2 \left( \left( {S_\mathrm{x}^- -2I+S_x^+ } \right) w_{i1} \right. \\ \left. +\left( {S_x^+ -S_x^+ S_\mathrm{y}^- -I+S_\mathrm{y}^- } \right) w_{i2} \right) =g_{i1} \\ \mu _1 w_{i2} -\mu _2 \left( \left( {S_y^+ -S_y^+ S_\mathrm{x}^- -I+S_\mathrm{x}^- } \right) w_{i1}\right. \\ \left. +\left( {S_\mathrm{y}^- -2I+S_y^+ } \right) w_{i2} \right) =g_{i2} \\ \end{array}} \right. . \end{aligned}$$
(29)

Based on the discrete Fourier transform, we have the following FFT properties for the shifting operators

$$\begin{aligned} { FS}_x^\pm f\left( {y_m ,y_l } \right)= & {} e^{\pm \sqrt{-1}Z_m }Ff\left( {y_m ,y_l } \right) , \end{aligned}$$
(30)
$$\begin{aligned} { FS}_y^\pm f\left( {y_m ,y_l } \right)= & {} e^{\pm \sqrt{-1}Z_l }Ff\left( {y_m ,y_l } \right) , \end{aligned}$$
(31)

where \(f( {y_m ,y_l })\) is the function in time domain, \(y_m \) and \(y_l \) are discrete frequencies. Let \(Z_m =\frac{2\pi }{N_1 }y_m , y_m =1,2,\ldots ,N_1 \) and \(Z_l =\frac{2\pi }{N_2 }y_l ,y_l =1,2,\ldots ,N_2 \), then (29) can be transformed into the following algebraic equations

$$\begin{aligned} \left( {\begin{array}{c@{\quad }c} {a_{11} }&{} {a_{12} } \\ {a_{21} }&{} {a_{22} } \\ \end{array} } \right) \left( {\begin{array}{cc} F( {w_{i1} } ) \\ F( {w_{i2} } ) \\ \end{array}} \right) =\left( {\begin{array}{cc} F( {g_{i1} } ) \\ F( {g_{i2} } ) \\ \end{array}} \right) . \end{aligned}$$
(32a)

where the coefficients are

$$\begin{aligned} a_{11}= & {} \mu _1 -2\mu _2 \left( {\cos Z_m -1} \right) , \nonumber \\ a_{12}= & {} -\mu _2 \left( {1-\cos Z_l +\sqrt{-1}\sin Z_l } \right) \nonumber \\&\left( {-1+\cos Z_m +\sqrt{-1}\sin Z_m } \right) , \nonumber \\ a_{21}= & {} -\mu _2 \left( {1-\cos Z_m +\sqrt{-1}\sin Z_m } \right) \nonumber \\&\left( {-1+\cos Z_l +\sqrt{-1}\sin Z_l } \right) , \nonumber \\ a_{22}= & {} \mu _1 -2\mu _2 \left( {\cos Z_l -1} \right) , \end{aligned}$$
(32b)

The determinant of this coefficient matrix is shown as follows, and more details can be referred to [29].

$$\begin{aligned} D=\mu _1^2 -2\mu _1 \mu _2 \left( {\cos Z_m +\cos Z_l -2} \right) . \end{aligned}$$
(32c)

This value is always positive for all discrete frequencies if \(\mu _1 >0\). Then, the discrete inverse Fourier transform can be used to update \(\vec {w}_i^k \) as follows

$$\begin{aligned} \left\{ {\begin{array}{l} \tilde{w}_{i1}^{k+1} =\mathfrak {R}\left( {F^{-1}\left( {\frac{a_{22} F\left( {g_{i1} } \right) -a_{12} F\left( {g_{i2} } \right) }{D}} \right) } \right) \\ \tilde{w}_{i2}^{k+1} =\mathfrak {R}\left( {F^{-1}\left( {\frac{-a_{21} F\left( {g_{i1} } \right) +a_{11} F\left( {g_{i2} } \right) }{D}} \right) } \right) \\ \end{array}} \right. . \end{aligned}$$
(33)

Finally, a simple projection technique can be used to guarantee the constraint \(|{\vec {w}_i }|=1\) to be satisfied if we define

$$\begin{aligned} \vec {w}_i^{k+1} =\frac{\vec {\tilde{w}}_i^{k+1} }{\left| {\vec {\tilde{w}}_i^{k+1} } \right| }. \end{aligned}$$
(34)

4.4 Minimization of the Auxiliary Variable \(v_i \)

For the minimization problem of (17) and (23), we can derive the Euler–Lagrange equation with respect to \(v_i \) while fixing the variables \(\varphi _i^{k+1} ,\vec {w}_i^{k+1} \) temporarily,

$$\begin{aligned} \beta \frac{v_i }{\left| {v_i } \right| }\delta \left( {\varphi _i^{k+1} } \right) +\lambda _{2i}^k +\mu _2 \left( {v_i -\nabla \cdot \vec {w}_i^{k+1} } \right) =0. \end{aligned}$$
(35)

This equation can be solved by using the analytical soft thresholding formula [19], which is given by

$$\begin{aligned} v_i^{k+1}= & {} Max\left( {\left| {\nabla \cdot \vec {w}_i^{k+1} -\frac{\lambda _{2i}^k }{\mu _2 }} \right| -\frac{\beta \delta \left( {\varphi _i^{k+1} } \right) }{\mu _2 },0} \right) \nonumber \\&\frac{\nabla \cdot \vec {w}_i^{k+1} -\frac{\lambda _{2i}^k }{\mu _2 }}{\left| {\nabla \cdot \vec {w}_i^{k+1} -\frac{\lambda _{2i}^k }{\mu _2 }} \right| }. \end{aligned}$$
(36)

4.5 Updating the Lagrange Multipliers

At the end of each step, we need to update the Lagrange multipliers \(\vec {\lambda }_{1i}^k \) and \(\lambda _{2i}^k \) after all sub-minimization problems have achieved their minimum, i.e.,

$$\begin{aligned} \vec {\lambda }_{1i}^{k+1}= & {} \vec {\lambda }_{1i}^k +\mu _1 \left( {\vec {w}_i^{k+1} -\nabla \varphi _i^{k+1} } \right) , \end{aligned}$$
(37)
$$\begin{aligned} \lambda _{2i}^{k+1}= & {} \lambda _{2i}^k +\mu _2 \left( {v_i^{k+1} -\nabla \cdot \vec {w}_i^{k+1} } \right) . \end{aligned}$$
(38)

In summary, all the proposed sub-problems can be solved effectively as discussed above. As in the alternating process, each sub-problem is easy to solve as other variables are kept as constants. As to the stopping criteria for each sub-problem, we will discuss them in next section.

5 Numerical Experiments

In this section, some numerical experiments on synthetic, and real images are presented to validate the performance and efficiency of proposed model and algorithm. All experiments are implemented using Matlab7.8 on a PC (Intel (R), CPU 2.60GHz). The proposed ADMM-P method will be compared with a traditional GDM [12]. For the purpose of fair comparison, the same initialization for both methods is used.

As described in [30], the iterations need to be terminated when the following criteria are satisfied:

(I) We need to monitor the following constraint errors in iterations:

$$\begin{aligned} \left( {{\begin{array}{ll} {R_{\vec {w}i}^k }&{} {R_{vi}^k } \\ \end{array} }} \right)= & {} \left( \left\| {\vec {w}_i^k -\nabla \varphi _i^k } \right\| _{L^{1}} \big /\left\| {\vec {w}_i^0 -\nabla \varphi _i^0 } \right\| _{L^{1}} \right. \nonumber \\&\left. \quad \left\| {v_i^k -\nabla \cdot \vec {w}_i^k } \right\| _{L^{1}} \big /\left\| {v_i^0 -\nabla \cdot \vec {w}_i^0 } \right\| _{L^{1}} \right) ,\nonumber \\ \end{aligned}$$
(39)

where, \(\Vert \cdot \Vert _{L^{1}}\) denotes the \(L^{1}\) norm on image domain \(\Omega \). All components in (39) are calculated point by point. If \(R_i^k <\varepsilon (\varepsilon \) is a small enough parameter) for \(i=1,2,\ldots ,n\), the overall iteration will be stopped. These good numerical indicators are also used to determinate the values of \(\mu _i ({i=1,2})\), which is the basis of penalty parameter adjustment.

(II) During iteration, the relative errors of Lagrange multipliers and the solution \(\varphi _i^k \) should be monitored. They should decrease to a sufficiently small level:

$$\begin{aligned} \left( {R_{\vec {\lambda }1i}^k }\quad {R_{\lambda 2i}^k } \right)= & {} \left( {{\left\| {\vec {\lambda }_{1i}^k -\vec {\lambda }_{1i}^{k-1} } \right\| _{L^{1}} }/{\left\| {\vec {\lambda }_{1i}^{k-1} } \right\| _{L^{1}} }}\right. \nonumber \\&\left. {{\left\| {\lambda _{2i}^k -\lambda _{2i}^{k-1} } \right\| _{L^{1}} }/{\left\| {\lambda _{2i}^{k-1} } \right\| _{L^{1}} }}\right) , \end{aligned}$$
(40)
$$\begin{aligned}&R_{\varphi i}^k =\left\| {\varphi _i^k -\varphi _i^{k-1} } \right\| _{L^{1}} /{\left\| {\varphi _i^{k-1} } \right\| _{L^{1}} }. \end{aligned}$$
(41)

Note that (40) can be quite small if the penalty parameters are large. This is due to the explicit dependency on the penalty parameters.

(III) The relative energy error can be chosen as stopping criterion:

$$\begin{aligned} R_e^k ={\left| {E^{k+1}-E^{k}} \right| }/{|{E^{k}} |}. \end{aligned}$$
(42)

where \(E^{k}\) is the energy value of (13a). The computation stops automatically when \(R_e^k \) is less than a predefined tolerance, which indicates that the energy approaches its steady state.

Until now, the presentation of the proposed ADMM-P is complete and we will show its effectiveness in next subsection with extensive experiments. Just keep in mind that all numerical quantities are plotted in log scale.

Next, we introduce some specific methods to tune parameters in the process of implementing the proposed approach. The two parameters in the NMS functional (1), \(\alpha \) and \(\beta \), control the length and curvature of the segmentation boundary. The ratio between \(\alpha \) and \(\beta \) is in relation to the connectivity and smoothness of the level lines. As discussed in [30], the connection of disconnected level lines and smoothness of level lines can be guaranteed by a larger parameter \(\beta \). In addition, another two parameters associated with Lagrange multipliers are in the augmented Lagrange energy functional (13): \(\mu _1 \) and \(\mu _2\), and they can be determined according to relative residuals (39), relative errors of the Lagrange multiplier (40), relative errors of \(\varphi _i^k\) (41) and the energy curve. One example is given to show such selection in Experiment 5.1.

5.1 Experiment on a Synthetic Image (Two Objects and One Background)

The first testing image is a synthetic image containing a bar and a U-sharp object (size \(128\times 128\)) as shown in Fig. 1a. In this example, two level set functions are required to describe the shapes. In order to speed up the evolution of contours, we initialize the two level set functions in Fig. 1b using the results from the piecewise constant image segmentation by using variational level set method. The final results of segmentation with depth are presented in Fig. 1c.

Fig. 1
figure 1

Segmentation with depth for a bar and a U-sharp object. a The original image and the results using traditional piecewise constant image segmentation model; b initialization for segmentation with depth; c segmentation results using both GDM and ADMM-P method

Full size image

In this experiment, the required parameters are selected as \(\alpha =0.5, \beta =3, \mu _1 =3000, \mu _2 =3, \varepsilon =3\), respectively, for the proposed ADMM-P algorithm. In order to determine the ordering relations of the bar and the U-shape object, we minimize the energy functional based on the assumptions that the U-shape object is occluded by the bar or the bar is occluded by the U-shape object. The results are listed in Table 1, from which we can deduce that the bar, the U-shape object and the background are ordered from the nearest to farthest with respect to the observer.

Table 1 Minimal energies of different ordering relations
Full size table

As to the case that the U-shape object is occluded by the bar, we illustrate the relative residuals (39), relative errors of the Lagrange multipliers (40), relative errors of \(\varphi _i^k \)(41) and energy curve in Fig. 2. The graphs come from Fig. 1c. From these plots, one can observe that the proposed algorithm has converged before 30 iterations. They also give an important clue about how to choose penalty parameters \(\mu _i ({i=1,2} )\). In order to guarantee convergence as well as the speed of convergence, the constraint errors \(R_{\vec {w}i}^k , R_{vi}^k ,R_{\vec {\lambda }1i}^k , R_{\lambda 2i}^k \) should converge steadily with nearly the same speed. If \(R_{\vec {w}i}^k , R_{vi}^k \) go to zero faster than others, then one can decrease \(\mu _i \) and vice versa. \(R_{\vec {w}i}^k , R_{vi}^k \) will converge to zero with the same speed as the iteration proceeds and the energy will decrease to a steady constant value when \(\mu _i \) are chosen properly.

Fig. 2
figure 2

The relative residual plots of a auxiliary variables; b Lagrange multipliers; c level set functions; d the energy functional

Full size image
Fig. 3
figure 3

Plots of decaying energy by GDM and ADMM-P methods

Full size image
Table 2 Number of iterations and time costs using GDM and ADMM-P methods
Full size table
Fig. 4
figure 4

Segmentation with depth for an image with one bar and two circles. a The original image and three regions via traditional image segmentation model; b the initializations for segmentation with depth; c the results via GDM; d the results via ADMM-P

Full size image

As to efficiency of the proposed algorithm, the energy curves produced by using the GDM [12] and the proposed ADMM-P are shown in Fig. 3 and the time costs are shown in Table 2. It is easy to see that the ADMM-P has faster convergence rate and higher efficiency. The parameters for the GDM method in this experiment are set as \(\alpha =0.5, \beta =3, \lambda =0.5, \varepsilon =3, \mathrm{d}t=5\times 10^{-4}\). By considering the comparisons of these two algorithms, we plot the energy functional of the original NMS model.

Table 3 Minimal energies of different ordering relations
Full size table
Fig. 5
figure 5

Comparison of energy curves obtained by GDM and ADMM-P

Full size image

5.2 Experiment on synthetic image (Three Objects and a Background)

The second testing image is a synthetic one with two circles and a bar (size \(128\times 128\)) shown in Fig. 4a. There are totally four regions to be segmented including three objects and a background. In this case, three level set functions are required. In Fig. 4b, the results by piecewise constant segmentation method are used as the initialization for the level set functions. The final results obtained by the GDM and ADMM-P are presented in Fig. 4c, d, respectively.

Table 4 Number of iterations and time costs using GDM and ADMM-P
Full size table
Fig. 6
figure 6

Segmentation with depth of a noise image. a The original image and the segmentation results via traditional image segmentation model; b the initializations for segmentation with depth; c the results of GDM; d final results of ADMM-P

Full size image

The parameters used in the proposed ADMM-P method are chosen as \(\alpha =0.3, \beta =2, \mu _1 {=}600, \mu _2 {=}3, \varepsilon =5\). In this experiment, there are \(3!=6\) possible combinations shown in Table 3 to be considered. After minimizing the energy functional based on the six assumptions separately, the ordering relations between the small circle, the big circle and the bar can be determined. From Table 3, we know that the correct ordering from the nearest to the most further is: small circle, big circle, bar and the background.

In the case of correct ordering, we present the plots of energy decay and time costs in Fig. 5 and Table 4, respectively, by using the GDM and ADMM-P methods in order to compare their computational efficiency. The parameters for the GDM method in this experiment are \(\alpha =0.3, \beta =2, \lambda {=0.1}, \varepsilon =3, dt=1\times 10^{-4}\).

Table 5 Minimal energies of different ordering relations
Full size table
Fig. 7
figure 7

Comparison of energy curves obtained by GDM and ADMM-P

Full size image
Table 6 Number of iterations and cost time using GDM and ADMM-P
Full size table
Fig. 8
figure 8

Segmentation with depth for a color image with a pencil and a hand. a The original image and two regions via traditional image segmentation model; b the initializations; c the results via GDM; d final segmentation results using ADMM-P; e evolution process of pencil; f evolution process of hand

Full size image

5.3 Experiment on a Noise Image (Two Objects and a Background)

The third experiment is conducted on a noise image (size \(100\times 100\)) with two circles (two objects and a background) as shown in Fig. 6a. The image is corrupted by Gaussian white noise with standard deviation 0.01. Figure 6b presents the initialization of the level set functions by using the results of piecewise constant segmentation method. The final results obtained by the GDM and ADMM-P are presented in Fig. 6c through Fig. 6d, respectively. It can be observed that the segmentation with depth is robust to noises.

Table 7 Minimal energies of different ordering relations
Full size table

In this experiment, the required parameters are selected as follows: \(\alpha =3,\beta =2,\mu _1 =4,\mu _2 =3,\varepsilon =3\) for the proposed ADMM-P method. For the ordering relations between the white circle and the gray circle, we minimize the energy functional based on the assumptions that the white circle occludes the gray circle and the gray circle occludes the white circle, respectively. From the results listed in Table 5, we can deduce that the correct ordering from the nearest to the farthest is: white circle, gray circle and the background.

In the case that white circle occludes gray circle, we further present the energy decaying trends and time costs in Fig. 7 and Table 6, respectively, obtained by using the GDM and ADMM-P methods in order to compare their computational efficiency. The parameters for the GDM method in this experiment are chosen as \(\alpha =3, \beta =2,\lambda {=0.05}, \varepsilon =3, \mathrm{d}t=5\times 10^{-4}\).

5.4 Experiment on a Color Image (Two Objects and a Background)

In the last experiment, a color image with a pencil and a hand (size \(256\times 242\)) shown in Fig. 8a is used. There is a pencil and a hand in this image, so two level set functions are required to represent the shapes. Figure 8b gives the initialization for the level set functions, and it is obtained by piecewise constant segmentation method. The final results obtained by the GDM and ADMM-P are presented in Fig. 8c, d, respectively. Figure 8e, f shows the process of evolution in details. It can be seen clearly that the pencil in the front keeps stable while fingers of the hand gradually connect behind the pencil. According to the segmentation model for vector-valued images proposed in [38], the averages of the data terms over all channels are used for coupling. In fact, the model used to solve color image segmentation with depth information can be stated as follows:

$$\begin{aligned} E= & {} \sum _{i=1}^n {\int _\Omega {\left[ {\alpha +\beta \left| {\nabla \cdot \left( {\nabla \varphi _i } \right) } \right| } \right] } \delta \left( {\varphi _i } \right) \mathrm{d}x} \nonumber \\&+\,\sum _{l=1}^m {\left\{ {\sum _{i=1}^n {\int _\Omega {\left( {f_l -c_{il} } \right) ^{2}H\left( {\varphi _i } \right) \prod _{j=1}^{i-1} {\left( {1-H({\varphi _j })} \right) } } \mathrm{d}x} } \right\} } \nonumber \\&+\,\sum _{l=1}^m {\int _\Omega {\left( {f_l -c_{\left( {n+1} \right) l} } \right) ^{2}\prod _{j=1}^n {\left( {1-H({\varphi _j })} \right) } \mathrm{d}x}} \end{aligned}$$
(43a)
$$\begin{aligned}&\hbox {s.t.}\quad |{\nabla \varphi _i }|=1 \end{aligned}$$
(43b)

where \(l=1,2,\ldots ,m\) denotes the number of layers of a vector-valued image.

Fig. 9
figure 9

Comparison of energy decreasing curves obtained by GDM and ADMM-P

Full size image
Table 8 Number of iterations and time costs using GDM and ADMM-P
Full size table

In this experiment, the parameters for the ADMM-P are selected as: \(\alpha =0.5,\beta =10,\mu _1 =85,\mu _2 =3,\varepsilon =3\). In order to determine the ordering relations of the pencil and the hand, we minimize the energy functional based on the assumptions that the pencil occludes the hand, or the hand occludes the pencil, respectively. From the results listed in Table 7, we can choose the correct ordering from the nearest to the farthest is: pencil, hand and the background.

In the case that the pencil occludes the hand, we compare the computational efficiency between the GDM and ADMM-P methods based on their energy decaying plots in Fig. 9 and the time costs in Table 8. The parameters for the GDM method in this experiment are \(\alpha =2, \beta =15, \lambda {=0.1}, \varepsilon =3, dt=8\times 10^{-5}\).

6 Conclusions

As it is known that the Nitzberg–Mumford–Shiota (NMS) model for image segmentation with depth can be described as the classical Mumford–Shah model, this model is computational expensive and time consuming. In order to solve this problem efficiently, for the variational level set formulation of the NMS, we first propose its equivalent simplified variational level set formulation by taking advantage of the property of level set functions as signed distance functions, then we develop the fast ADMM (alternating direction method of multipliers) projection method by combining the ADMM method and projection method. Thus, the original complicated problem is decomposed into a series of simple sub-problems of optimization, and each sub-problem can be easily solved accordingly. Extensive experiments have validated the effectiveness of the proposed ADMM-P approach as the ADMM-P method is much faster than the traditional algorithms based on the GDM and this may be due to the merits of the Gauss–Seidel method, soft thresholding formulas, FFT method, projection method as well as model reduction. Also, the strategies of model simplification and constraint projection can be easily extended to other variational level set models with/without curvatures.

In this paper, the number of objects in an image is known and in future, we will investigate this problem without this assumption.