Keywords

1 Introduction

Image segmentation is a key issue in image processing, pattern recognition and computer vision. It is to make full use of the color, brightness, texture and shape feature information. Through image segmentation, the image is divided into a plurality of small areas. So that, image has some identical or similar characteristics in the same region, and has larger differences between different regions. Contour extraction can be understood as the image region of interest to the user to be split, and then extract the contour of the object target. The accuracy of the segmentation results directly affects image analysis and contour extraction. However, because of the influence of illumination, the reflection of object’s surface and other aspects, the results of segmentation often appear deviation. Thus, the segmentation of non-uniform illumination on image has become an important research topic in recent years.

For various methods of image segmentation, people put forward different classifications from different view. According to two basic properties of the image– non continuity and homogeneity [1], people can realize image segmentation. Non continuity refers to a sudden change in gray level, so it can be detected by dot, line and side to complete image segmentation. Homogeneity is refers to the similarity in the region of the gray level, so it can be cut by thresholding, clustering, region growing and region merging and splitting method to realize segmentation. Some methods are presented by the literature [2]. Using these methods, the image can be segmented based on the image space method, feature space method and the physical characteristics method, Lucchese has made very good summary in the literature [2].

Traditional image segmentation methods, such as threshold, edge detection, region growing, splitting and merging, although simple, shortcut, but poor robustness, are easy to leak in weak point of boundary. The pertinence of these methods is strong. With the rapid development of computer technology, people not only hope to be able to develop segmentation methods more efficient, robust, but also hope to find a general unified framework for these methods.

In recent years, people put forward the framework of image segmentation based on energy minimization. The framework can fuse the upper visual information and context information, in order to get better segmentation of interesting targets. Its basic feature is to construct an energy function, then according to the energy function, select the appropriate optimization technology, which makes the function minimization. According to the continuity of energy function method based on the framework, image segmentation methods can be divided into two categories [3].

  1. (1)

    The optimization is defined on the functional of a continuous contour or surface. This kind of method is a continuous functional to infinity in space R. It uses the variational method, gradient descent method and other optimization technology to optimize energy function. The basis of its value calculation is mainly the finite difference, finite element method. The method mainly comprises the snake, geodesic active contour model [4–10], regional competition and some segmentation method based on level set method [11–15]. Its characteristics are: The continuous functional integrates region and boundary information to segment the target, some properties of these information are formed by the geometric expression [16, 17]. Optimization technology of the method can only ensure that local optimal variable values for the energy function, cannot get the global optimization.

  2. (2)

    The cost function is defined on a discrete variable set optimization. This kind of method is directly transformed the problem into a combinatorial optimization in finite dimensional integer space, and using dynamic programming, maximum flow minimum cut algorithm to optimize. The methods mainly include the path based method and graph cuts method. The path based method uses dynamic programming to calculate the optimal path, such as intelligent scissors [18] and live-wire [19] using the Dijkstra algorithm, DP-snake [20] algorithm using Viterbi algorithm. Its characteristics are: the boundary information can be naturally expressed very well, but it is difficult for regional information expression [21]. Because the three dimensional goal cannot be used to describe the path, so the segmentation problem was defined in the two dimensional space.

The method based on graph cuts can be very good to overcome the above shortcomings, not only the problem can be extended to the N dimension space, but also provides a general cost function. Region and boundary information and some topological restrictions can naturally add into the cost function. The graph cuts theory is introduced into the field of computer vision for the first time by Yuri Boykov and Maric-Pierre Jolly [22, 23]. They put forward and realized a method that using the max-flow/min-cut algorithm [24] to minimize the energy function and then gain the contour of object. Many researchers proposed different segmentation methods for the shortcoming of graph cuts algorithm. Rother et al. put forward GrabCut algorithm [25], which is better currently a method to extract the target. Liu et al. propose a method of interactive graph cuts in [26]. Jain et al. extract object contour by predicting sufficient annotation strength model based on graph cuts theory [27]. Literature [28] use super-pixel Gaussian model to build graph cuts model to improve the efficiency of the graph cuts algorithm. Wang et al. combine fuzzy C-means clustering algorithm with graph cuts to segment image [29]. Liu et al. combine graph cuts theory and wavelet transform algorithm to improve the efficiency and quality of the graph cuts segmentation and improve shrinking bias phenomena that occur during the extraction [30]. Xu et al. proposed an active contour algorithm based on graph cuts (GCBAC) theory since the traditional active contour algorithm is easy to fall into local optimum [31]. Zhang et al. put forward active contour edge method based on conjugate gradient B-spline [32]. Zheng et al. use active contour algorithm based on graph cuts to achieve the selective extraction of local and global contours [33].

Based on the work mentioned above, we present a novel contour extraction method which merges photometric information into the energy function, and construct a new energy function. The photometric information of pixels in the image is expressed with the mean values of the brightness information and color information of the image. The gray values of pixels are replaced by the photometric information of pixels. This method uses the maximum flow/minimum cut algorithm to enable initial contour converge to the target profile, finally extracts global contour of image object. Then using partial segmentation method, the average values of luminosity in each local area of the image are obtained. Adjusting the value of illumination, the appropriate threshold of photometric in an area of image is selected. Against the problem of the target surface of image with different brightness, it can be assigned with different photometric thresholds in different surface, and can achieve the accurate local contour of the subject image with little light on it. The method is based on active contour model and graph cuts algorithm, can overcome the shortcomings occurred in previous literatures. This algorithm integrates luminosity information into the energy function, expands the scope of application. The experiment results demonstrate that the method can extract the global contour and local contour selectively more accurately and faster for more images.

2 Basic Theory and Method

2.1 Basic Theory of Active Contours

We know that C is a closed subset in Ω which is made up of a finite set of smooth curves, if Ω i denotes the connected components of Ω\C, then the segmentation problems in computer vision can be defined as follows: give an image I 0, find a decomposition Ω i of Ω and an optimal piecewise smooth approximation I of I 0, such that I varies smoothly with each Ω i, and rapidly or discontinuously across the boundaries of Ω i. So the curve evolution formulation based on active contours model can be represented as follow:

$$ \begin{aligned} E_{ACM} (I,C) = &\iint_{\Omega } {(I - I_{0} )^{2} dxdy} \hfill \\ &\qquad \!\!+ \mu \iint_{\Omega \backslash C} \;{|\;\nabla I\;|^{2} dxdy} \hfill \\ \end{aligned} $$
(1)

Where I 0 is the initial image on Ω to be segmented, I is an piecewise smooth approximation image of I 0 with discontinuities on interface curve C, |C| is the length of C, μ is a positive parameters which control the amount of smoothness.

2.2 Basic Theory of Graph Cuts

The main idea of graphic cuts is: the image is mapped to a network diagram with terminal nodes. Each adjacent pixel in the image is corresponded to the network node in the graph. The weight of edge in the network diagram shows the similarity of adjacent pixels. According to this weight, a suitable energy function is designed. By minimizing the energy function, the image segmentation can be finished, finally achieve the contour of object.

Setting G = (V, E) is an undirected graph. Where, V is a set of vertices, E is a set of edges. In the set of vertices V, connection (x, y) is called an edge e, which have two different directions, i.e., from x to y and from y to x, respectively. It is denoted by (e, x, y) and (e, y, x). Every edge of the network graph G is obtained using this operation. Thus the edge obtained is denoted as E, and define a capacity function c in E. Then graph G and capacity function c form a network is denoted by N = (G, s, t, c). Were s, t are called source and sink. A function Φ: E → R is called a stream of network which meets the following conditions:

  1. (1)

    Φ(e,x,y) = -Φ(e,y,x), for any(e,x,y)∈E(x ≠ y);

  2. (2)

    Φ(x,v) = ∑(e,x,y) ∈EΦ(e) = 0, for any x∈V\{s,t};

  3. (3)

    Φ(e) ≤ c(e), for any e∈E.

In this case, Φ (x, v) = Σ (e, x, y) ∈EΦ (e) is called flow for the stream Φ. According to Ford-Fulkerson maximum flow/minimum cut theorem, network diagram can obtain the minimum cut by calculating the maximum flow of the network.

3 Construction of New Model

3.1 Photometric Model Establishing

The surface brightness and color rendering are determined by surface radiation to gaze into the eyes. A mathematical model is built to simulate the phenomenon of surface illumination. According to mathematical model, through calculating the surface brightness of radiation to gaze into the eyes, we can get the color and brightness of each pixel in the image.

When light arrives the surface, only the transmitted light and reflected light can be seen by the human’s eyes, and produce visual effects, the light absorbed by the object is converted into heat, so the last object rendered in color and brightness is controlled by the transmitted light and reflected light. If the object is opaque, with the absence of transmitted light, the color of the object is only determined by the reflected light. In light model, simple lighting model considers only the reflective case of the surface under the direct rays of light. Simple lighting model is expressed as:

$$ I = I_{e} + I_{d} + I_{s} $$
(2)

In (2), I expresses that the light intensity of the surface of the object reflect to the viewpoints; I e indicates the intensity of ambient light; I d represents the intensity of diffuse light and I s expresses light intensity of specula reflected.

There are many representations of the color image, such as RGB, color format YCbCr, LAB and HSB. In color format YCbCr, Y represents the image brightness, Cb, Cr represent blue component and yellow component of an image, which represent color degrees of pixel. Integrated color image consists of the three components, the average value of which represented the photometric value of the pixel.

First, we consider the image represented with RGB form, the individual components of YCbCr for each pixel can be obtained using the following equations.

$$ \left\{ \begin{aligned} & Y = 0.257 \times R + 0.504 \times G + 0.098 \times B + 16 \\ & Cb = - 0.184 \times R - 0.291 \times G + 0.493 \times B + 128 \\ & Cr = 0.439 \times R - 0.368 \times G - 0.071 \times B + 128 \\ \end{aligned} \right. $$
(3)

Then, L i represents the photometric value of the pixel I,

$$ L_{i} = (Y_{i} + Cb_{i} + Cr_{i} )/3 $$
(4)

Where, Y i, Cb i and Cr i respectively represent lightness, blue component and red component of the pixel i.

3.2 Global Contour Model

Graph cuts is a powerful combinatorial optimization method. It can solve discrete labeling problems by minimizing energy function. The model can be written in the following standard form:

$$ E(f) = \sum\limits_{p \in P} {D_{p} } (f_{p} ) + \sum\limits_{(p,q) \in N} {V_{p,q} } (f_{p} ,f_{q} ) $$
(5)

Where, f is a labeling function from set of pixels P map to set of labels L. N is a neighborhood system. The term D p(f p) is used to measure the penalty of assigning label f p to pixel p. The term V p,q(f p,f q) is used to measure the penalty of assigning labels fp, fq to the pixels p and q.

In this paper, we want to use graph cuts method to optimize the active contour model. The goal is to find a labeling which can minimize the energy function like (5). Now based on continuous active contour model, we rewrite the energy function (1) in discrete form as follows:

$$ E = \mu \times E_{smooth} (p,q) + E_{data} (p) $$
(6)

Where, μ is the regularizing parameter. \( E_{smooth} (p,q) \) is used to measure the extent to which pixels (p,q) are not piecewise smooth. \( E_{data} (p) \) is used to measure the disagreement between pixel p and the observed data.

Taking into account the impact caused by the photometric value of the image, the final contour of object should be inaccurate, and existing methods only consider removing the shaded area to get the overall outline of the image. This paper attempts to integrate photometric information into energy function based on active contour model and graph cuts, in order to improve the method proposed in literature [33] about splitting the images affected large by light. Since the regularizing parameter μ in energy (6) is used to balance the edge and region terms. It often influences the segmentation results. In order to simplify the problem, we set the regularizing parameter μ as a constant value 1, then energy function can be written as:

$$ E = E_{smooth} (p,q) \times E_{data} (p,q) $$
(7)

Where, E data (p,q) indicates the penalty for assigning pixels p and q different labels.

$$ \begin{aligned} E_{smooth} (p,q) & = \sum\limits_{p,q \in \Omega } {\sum\limits_{p \in N(q)} {\frac{{\omega_{pq} \times ((1 - x_{p} )x_{q} + x_{p} (1 - x_{q} )}}{{1 + \beta \left| {I(p) - I(q)} \right|}}} } \\ E_{data} (p,q) & = \sum\limits_{p} {Min((I_{ps} } + I_{qt} ),(I_{pt} + I_{qs} )) \\ \end{aligned} $$
(8)

Where, β > 0 is a scaling parameter, pixel p is in the vicinity of pixel q, i.e. N(q), N denotes a 4-connected neighborhood, and ω pq  = σ 2 Δθ pq /∣e pq ∣ with the angular differences Δθ pq between the nearest edge lines and the length∣e pq ∣ of edge epq. I ps, I qt, I pt, and I qs can be calculated as follow

$$ \begin{aligned} I_{ps} & = (I(p) - c_{s} )^{2} ;I_{qt} = (I(q) - c_{t} )^{2} \\ I_{pt} & = (I(p) - c_{t} )^{2} ;I_{qs} = (I(q) - c_{s} )^{2} \\ \end{aligned} $$
(9)

In (9), c s and c t can be calculated by formula (10) based on the corresponding relations in Table 1. Figure 1 shows the RGB color value corresponding to the pixel position in Table 1. I (p) is the photometric information of the pixel p:

Table 1. The conversion process of color pixel values
Full size table
Fig. 1.
figure 1

RGB color value corresponding to the pixel position number (Color figure online)

Full size image
$$ \begin{aligned} c_{s} & = (\sum\limits_{p} {I(p)(1 - x_{p} } ))/(\sum\limits_{p} {(1 - x_{p} } )) \\ c_{t} & = (\sum\limits_{p} {I(p)x_{p} } )/(\sum\limits_{p} {x_{p} } ) \\ \end{aligned} $$
(10)

According to Table 1 and formula (7), the network graph can be constructed. The weights of edges connecting neighboring pixel nodes are assigned as E smooth(p,q) × E data(p,q); for terminals S/T links, if the following definition for each pixel p is true

$$ I_{ps} > I_{pt} + \;|\;I_{ps} - I_{pt}\; |\;/2 $$
(11)

Then add link pT with a big weight. if the following definition for each pixel p is true

$$ I_{pt} > I_{ps} + \;|\;I_{ps} - I_{pt} \;|\;/2 $$
(12)

Then add link Sp with a big weight.

3.3 Local Contour Model

In general, we conduct the research to the image, not on the entire image, but further on a part of an image. Therefore, the research on local image segmentation has more practical significance. We study on the image of the object surfaces with different luminosities, each face contour can be extracted by changing photometric values of image intensity. From previous analysis, we know that every pixel value of local area in the image can be calculated as follow:

$$ \begin{aligned} c_{s} (\phi) & = \frac{{\int_{\Omega } {u_{0} (x,y)H_{\varepsilon } (\phi(x,y))dxdy} }}{{\int_{\Omega } {H_{\varepsilon } (\phi(x,y))dxdy} }} \\ c_{t} (\phi) & = \frac{{\int_{\Omega } {u_{0} (x,y)(1 - H_{\varepsilon } (\phi(x,y)))dxdy} }}{{\int_{\Omega } {(1 - H_{\varepsilon } (\phi(x,y)))dxdy} }} \\ \end{aligned} $$
(13)

To obtain surface contour extraction in the image based on photometric information, photometric value is as follow.

$$ L(i,j) = \left\{ {\begin{array}{*{20}l} {c_{s} } & {L(i,j) > c_{thl} } \\ {c_{t} } & {otherwise} \\ \end{array} } \right. $$
(14)

Where, (i, j) represents image position of the pixel. c thl represents threshold value of the c s . L(i, j) represents photometric value of the image pixel in the position (i, j).

After converted graph cuts format, every average photometric value of local area can be calculated as follow:

$$ \begin{aligned} c_{ls} & = (\sum\limits_{p} {L(p)(1 - x_{p} } ))/(\sum\limits_{p} {(1 - x_{p} } )) \\ c_{lt} & = (\sum\limits_{p} {L(p)x_{p} } )/(\sum\limits_{p} {x_{p} } ) \\ \end{aligned} $$
(15)

Then use the following local energy equation to extract evolution contours.

$$ E_{l} = E_{lsmooth} \left( {p,q} \right) \times E_{ldata}^{{}} (p,q) $$
(16)

Where E lsmooth and E ldata can be calculator as following

$$ \begin{aligned} E_{lsmooth} (p,q) & = \sum\limits_{p,q \in \Omega } {\sum\limits_{p \in N(q)} {\frac{{\omega_{pq} \times ((1 - x_{p} )x_{q} + x_{p} (1 - x_{q} )}}{{1 + \beta \left| {L(p) - L(q)} \right|}}} } \\ E_{ldata} (p,q) & = \sum\limits_{p} {Min((L_{ps} } + L_{qt} ),(L_{pt} + L_{qs} )) \\ \end{aligned} $$
(17)

L ps, L qt, L pt, and L qs can be calculated as following

$$ \begin{aligned} L_{ps} & = (L(p) - c_{ls} )^{2} ;L_{qt} = (L(q) - c_{lt} )^{2} \\ I_{pt} & = (L(p) - c_{lt} )^{2} ;L_{qs} = (L(q) - c_{ls} )^{2} \\ \end{aligned} $$
(18)

For terminals S/T links, we have the following definitions

$$ L_{ps} > L_{pt} + \;|\;L_{ps} - L_{pt}\; |\;/2 $$
(19)

And

$$ I_{pt} > I_{ps} + \;|\;I_{ps} - I_{pt} \;|\;/2 $$
(20)

For each pixel p, if the definition (19) is true and xp is 1, then add link pT with a big weight. if the definition (20) is true and xp is 0, then add link Sp with a big weight.

4 Experimental Results and Analysis

With the above algorithm, we will respectively carry on experiments with light affected on different initial contour curves. Experimental platforms are as follow. Operating system is Win7, development tool is Matlab R2012a. Main parameters of computer hardware are CPU: Intel (R) Core (TM) i3, RAM: 2G. On these platforms, four color images are used to test global contour extraction. Two color images are used to test local contour extraction.

4.1 Global Contour Extraction

Four color images are illustrated in Fig. 2. First row is the process for synthetic image, second row is the process for lotus image, third row is the process for sand pot image, fourth row is the process for stone image. The images in Fig. 2(a) and (b) represent the original image and the initial contour line. Figure 2(c) and (d) are respectively the results of the original algorithm for image not affected by light and the proposed algorithm for image affected by light.

Fig. 2.
figure 2

The comparison of segmentation result for illumination influence image

Full size image

Table 2 is the effect comparison of segmentation for illumination influence images. It shows that the perimeter of the target is represented by the number of pixels of the contour line, and the area of the target is represented by the number of pixels in the contours, with which to compare the effect of segmentation. The target perimeter and area obtained by proposed method are close to the original from the data in Table 2. It can be seen from Fig. 2 also, the contour obtained by the original method is not the exact contour of the object, but also part of the target.

Table 2. The effect comparison of segmentation for illumination influence images
Full size table

Table 3 is the efficiency comparison of segmentation for illumination influence images in the above experiments. The ratio of the target perimeter and area of the original algorithm and the proposed method against that of the desired target, represents the results of image segmentation. The time in Table 3 represents the segmentation efficiency. It can be seen from the image called stone in Fig. 2, the more complex the background image, the more extraction time. It can be seen from the data of Table 3 that, the results obtained from proposed method are close to the ideal result, and the error is about 1 %.

Table 3. The efficiency comparison of segmentation for illumination influence images
Full size table

It can be seen from Tables 2 and 3 that the results of two algorithms are similar to the ideal result. In contrast, the errors of image called lotus is large, so the error is large for more complex background image although contour of an object can be obtained; It can be seen from the time of extraction in Table 3 that the speed of the convergence of the proposed method is significantly better than the original method. From the data of the image named synthetic image in Fig. 2 can be seen that, it can quickly converge to the target profile for the images with the proposed method. The time of extraction is longer for larger image. Column 3 and column 4 in Table 3 are results of time comparison of four images of the object contour with two methods. The extraction times of four images are placed in two columns, the column 3 is the extraction time used in original method, and the column 4 is the extraction time used in proposed method. From the results you can clearly see that the efficiency of proposed method is better than the original method.

4.2 Local Contour Extraction

The experiments were performed on two images as shown in Fig. 3. In order to facilitate the expression of the image, we give different names for each surface in Fig. 3. Stone image in Fig. 3(a), bright object, dark side, and shadows are labeled as surface A, surface B and surface C separately; polyhedral image in Fig. 3(b), from top to bottom, from left to right, the surfaces are labeled as surface A, surface B and surface C separately.

Fig. 3.
figure 3

The images to be processed

Full size image

Using the formula (13), we can get different average photometric values as target surface calculation results of foreground and background values. As shown in Table 4, shows the average photometric value of the local surface in different images, the foreground average brightness value C s and background average brightness value C t.

Table 4. The average photometric value of the local surface
Full size table

Now on the average photometric values obtained in the previous section analysis, we take the name stone image in Fig. 3(a) as an example to conduct a detailed analysis.

Figure 4 is a process of local contour extraction for the stone image in Fig. 3(a). From Table 4, we can get some results just like following. When using surface A as target, the average photometric value of surface A is 171.799, the average photometric value of other remaining part of the surface is 168.095. When the surface B for the prospects, the average photometric value of surface B is 142.020, the average photometric value similarly in other parts of the surface is 137.828. The average photometric value of surface C is 141.967, the average photometric value of the other part of the surface is 134.755. Now to obtain the surface A in the image contour extraction based on photometric information, the image of the photometric value is as follow according to the previous analysis.

Fig. 4.
figure 4

Local contour extraction for the stone image

Full size image
$$ L(i,j) = \left\{ {\begin{array}{*{20}l} {171.799} & {L(i,j) > 170} \\ {168.095} & {otherwise} \\ \end{array} } \right. $$
(21)

Where, (i, j) represents the pixels in the image position; L (i, j) represents (i, j) pixel photometric value.

Because the photometric values of surface B and surface C are close, so need for two groups of photometric data to change a pixel value, and then extract the contours. The photometric value of surface B and C can be obtained respectively according to following equations.

$$ L(i,j) = \left\{ {\begin{array}{*{20}l} { 1 3 4. 7 5 5} & {L(i,j) < 1 3 7. 8 2 8} \\ { 1 4 1. 9 6 7} & {otherwise} \\ \end{array} } \right. $$
(22)
$$ L(i,j) = \left\{ {\begin{array}{*{20}l} { 1 4 2. 0 2 0} & {L(i,j) > 141.967} \\ { 1 3 7. 8 2 8} & {otherwise} \\ \end{array} } \right. $$
(23)

According to the process of local contour extraction of the pictures above, the extraction results are shown in Fig. 4. First row is the process of local contour extraction for surface A. Second row is the process of local contour extraction for surface B. Third row is the process of local contour extraction for surface C.

Figure 5 is the result of partial contour extraction of the image polyhedron. As can be seen from the figure, the proposed method may be the local contour extraction image accurately. Because the map polygon A has three surface photometric values, but they are very closely, so in the contour extraction, can obtain one surface contour. Surface B and surface C have different photometric values, so can accurately obtain the contour lines separately. In Fig. 5, first row is the process of local contour extraction for surface A. Second row is the process of local contour extraction for surface B. Third row is the process of local contour extraction for surface C. Therefore, the difference of the image photometric value is more big, the proposed method can obtain more accurate local contour.

Fig. 5.
figure 5

Local contour extraction for the polyhedron image

Full size image

The photometric values of other pictures can be processed using same method. The contour extraction of the processed image can be done using previous method.

5 Conclusions

In this paper, photometric information is integrated into the energy function, with active contour model based on graph cuts to extract contour of the object. We can see that it has certain constraints for complex background images in global division. Therefore, to improve the method again, and expand the use of the method, the local division is used in the next contents of this paper. The method can extract contours of selective part of the image, such as bright side part, the shaded part and the dark side part. It can be seen from the results that regardless of the image large or little affected by light, the proposed method can obtain more accurate contour, and the proposed algorithm is better than the original algorithm in convergence time.