Color and multiscale texture features from vectorial mathematical morphology

Abstract

In this paper, we explore an original way to compute texture features for color images in a vector process. Using a dedicated approach for color ordering, we produce a complete framework for color mathematical morphology adapted to human visual system characteristics. Then, morphological multiscale texture features are defined. To understand the texture feature behavior, we present the feature response to basic images variations. Finally, we compare the texture feature performance in front of a classical classification task using Outex database.

1 Introduction

Mathematical morphology owns to the nonlinear image processing family and is well suited for texture analysis using the pattern spectrum descriptors. Inside these feature analysis tools, the initial expressions allow to define three groups of texture features: the pattern spectrum, the morphological covariance and the fractal signature. The pattern spectrum and the fractal signature characterize the distribution of an object count depending on their size and their inter-distance. The morphological covariance characterizes the repetition of a pattern based on a given distance and direction. One of the main interests of morphological spectra is to be easily understandable by the user. With this paper, our aim is to recall and explain the behavior of these morphological texture features and the generic adaptation to the color domain using color distance functions.

As few texture features are used, we chose to start explaining the expected spectra depending on the nature of texture descriptors and on the variations of simple pattern for grayscale images (Sect. 2). Then, we will describe the mathematical construction of the morphological texture descriptors, and we describe the processed responses to the basic texture variations (Sects. 2.2–2.4). The second half of the paper is dedicated to the color adaptation of these texture features. The distance-based color ordering scheme of the morphological process is developed in Sect. 3. Experiments based on fractal dimension assessment [1] having shown the accuracy gain the distance-based ordering scheme used in front of other ordering approaches [2, 3]. Thanks to this ordering construction and to the mathematical morphology framework, the morphological texture features can be processed for color images in a vector way. Results are first detailed for three texture images. Finally, a classification process is used to assess the performance of the texture features comparing obtained results in grayscale or color domain (Sect. 4).

2 Morphological spectrum

Morphological spectra come from the initial work of Matheron [4] for granulometric analysis of material samples. The asked question is how to analyze the ratio between the particle count and the particle size taking into account the object shape. Mathematical morphology offers an elegant way to assess relationships between object size and their count in a multi-iteration process. Initial work that was expressed for binary images allowed to define the first pattern spectra. Such spectra can be interpreted as the concatenation of two distributions: the first depending on the objects count at a given size and the second depending on the count of non-object (background), so the inter-object distance.

Figure 1b presents the expected pattern spectrum for the binary image of Fig. 1a. The square image includes 16 squares of $25\times 25$ pixels, separated by 15 pixels. Intuitively, the expected spectrum must be defined by two Dirac functions, first one at a location corresponding to the square size and second one at a location corresponding to the inter-distance size. Then, the Dirac function magnitude must be correlated to the object count.

Fig. 1

Expected spectrum has two Dirac functions: first for the squares (squares of $25\times 25$ pixels) and second for background (distance of 15 pixels)

Working only on the relationship between size distribution of particle in both classes “background” and “objects” can be done intuitively using binary images. Gray-level extension, using lattice theory, includes naturally the ordering notion and the maximum and minimum extraction in a given neighborhood [5, 6]. In addition, the object counting process is transformed in a generic volume measurement, allowing to embed the object and background variations. In this case, the main interest of the morphological process is to be differential and then invariant to variations of gray-level intensity average. Morphological texture features inherit this average variation invariance, keeping only the contrast magnitude between objects and background.

2.1 Expected theoretical spectra for grayscale images

Before to compare the morphological spectrum definition and results, we propose to define the spectrum response of morphological texture features to simple content variations. The initial texture images are composed by a set of $n$ squares of size $d$, separated by $e$ pixels, and with a contrast of $h$. The image size is preserved ($325\times 325$ pixels) and selected to avoid impact between objects and border inside the iterative process. The image background is fixed to 0 (black).

Variations in image content are obtained changing different parameter: object count, size, contrast or distance. The size $d$ of squares varies from $5\times 5$ (Fig. 2a) to $25\times 25$ pixels (Fig. 2e). The inter-object distance $e$ varies from $5\times 5$ (Fig. 2f) to $25\times 25$ pixels (Fig. 2j). The contrast¹ $h$ between grayscale value of background and grayscale value of objects is modified with almost identical steps. To modify the contrast, we evolve the grayscale value of object from 20 (Fig. 2k) to 100 (Fig. 2o). The number of squares per line² varies from 2 (Fig. 2p) to 6 (Fig. 2t). Caption color in Fig. 2 corresponds to curve color in Figs. 3 and 5.

Fig. 2

Squares undergoing different spatial and grayscale modifications. The grayscale value of background is fixed

Fig. 3

Expected spectra depending on different variations of images of Fig. 2 for spectra characterizing the pattern size and distance in images (pattern spectrum and fractal signature)

Fig. 4

Expected spectra depending on different variations of images of Fig. 2, for spectra characterizing the pattern repetition in an image

Fig. 5

Morphological spectra obtained for figures of squares (Fig. 2)

Under a binary approximation, pattern spectra characterize pattern distribution among their size and distance, without constraint on the spatial texture regularity. The gray-level extension transforms the counting problem into a volume measurement among the scales (i.e., iterations) combining in the same integral the area and the contrast measurements. Mathematical covariance is similar to an autocorrelation function, analyzing the pattern repetition at a certain distance and direction. Fractal signatures assess the volume measure evolution using a morphological dilation/erosion to remove small object parts at each scale. So fractal signature and pattern spectrum have similar behavior. Then, we reduce our explanation of morphological texture features to pattern spectrum and morphological covariance.

Figure 3 presents the different expected pattern spectrum responses to texture variations. To simplify the following explanation, we call the Dirac function characterizing the objects $D_\mathrm{obj}$ and $D_\mathrm{back}$ for the background. When the object size “d” is modified (Fig. 3a): $D_\mathrm{obj}$ is shifted on the abscissa axis depending on the size value; $D_\mathrm{obj}$ magnitude remains unchanged; $D_\mathrm{back}$ is not shifted; $D_\mathrm{back}$ magnitude decreases as the objects size increases and consequently the background area decreases. When the inter-object distance “e” is modified (Fig. 3b): $D_\mathrm{obj}$ is not shifted; $D_\mathrm{obj}$ magnitude remains unchanged; $D_\mathrm{back}$ is shifted on the abscissa axis depending on the distance value; $D_\mathrm{back}$ increases when the inter-object distance $e$ grow up, due to the increasing background area between objects. When the contrast “h” is modified (Fig. 3c): $D_\mathrm{obj}$ is not shifted; $D_\mathrm{obj}$ magnitude increases as the grayscale contrast increases; $D_\mathrm{back}$ is not shifted; $D_\mathrm{back}$ magnitude remains unchanged (same background area). When the object count “n” is modified (Fig. 3d): $D_\mathrm{obj}$ is not shifted; $D_\mathrm{obj}$ magnitude increases as the object count increases; $D_\mathrm{back}$ is not shifted; $D_\mathrm{back}$ magnitude decreases as object count increases (decreasing background area).

For the morphological covariance, we need to take into account the pattern repetition at a certain distance and orientation. Soille, then Aptoula proposed to analyze the texture using four orientations: 0$^\circ$, 45$^\circ$, 90$^\circ$ and 135$^\circ$ [3]. As the proposed texture is constructed with aligned squares among rows and columns, the morphological covariance for the rotation equal to 0$^\circ$ and 90$^\circ$ is identical as well as the results for the rotations equal to 45$^\circ$ and 135$^\circ$. As previously defined, to simplify the following explanation, we characterize the maximum obtained for the spectra by a Dirac function, and we call them $D_{0^\circ \& 90^\circ }$ and $D_{45^\circ \& 135^\circ }$.

When the object size “d” is modified (Fig. 4a): The Dirac $D_{0^\circ \& 90^\circ }$ is shifted on the abscissa axis depending on the size value (as the distance between the edges of squares remains unchanged, the distance between the square centers varies depending on the square size); $D_{0^\circ \& 90^\circ }$ magnitude increases as the object size increases; $D_{45^\circ \& 135^\circ }$ is shifted on the abscissa axis depending on the size value; $D_{45^\circ \& 135^\circ }$ magnitude increases as the object size increases. When the inter-object distance “e” is modified (Fig. 4b): $D_{0^\circ \& 90^\circ }$ is shifted on the abscissa axis depending on the distance value; $D_{0^\circ \& 90^\circ }$ magnitude remains unchanged; $D_{45^\circ \& 135^\circ }$ is shifted on the abscissa axis depending on the distance value; $D_{45^\circ \& 135^\circ }$ magnitude remains unchanged. When the contrast “h” is modified (Fig. 4c): $D_{0^\circ \& 90^\circ }$ is not shifted; $D_{0^\circ \& 90^\circ }$ magnitude increases as the contrast increases; $D_{45^\circ \& 135^\circ }$ is not shifted; $D_{45^\circ \& 135^\circ }$ magnitude increases as the contrast. When the object count “n” is modified (Fig. 4d): $D_{0^\circ \& 90^\circ }$ is not shifted; $D_{0^\circ \& 90^\circ }$ magnitude increases as the object count increases; $D_{45^\circ \& 135^\circ }$ is not shifted; $D_{45^\circ \& 135^\circ }$ magnitude increases as the object count increases. In each previous cases, magnitude of $D_{45^\circ \& 135^\circ }$ is lowest than $D_{0^\circ \& 90^\circ }$ because the count of object pixels of same values is less important after a spatial shift along the diagonal axis than after a spatial shift along the horizontal/vertical axis.

Thanks to this generic knowledge, we can develop the mathematical expressions and analyze their responses facing the expected spectra.

2.2 Pattern spectrum

Pattern spectrum is a shape-size descriptor developed by Matheron [4]. Texture analysis is processed using successive morphological opening ($\gamma$) for the granulometric part of the spectrum and closing ($\varphi$) for the anti-granulometric part. These iterative morphological transforms delete objects from smallest to largest ones and the equivalent for the background parts (if we kept the initial ways to explain the mathematical construction, even if these concepts are little bit more complex for gray-level images). Consequently, the pattern spectrum $PS$ (Eq. 1) of image $f$ is the object distribution depending on $g$ the structuring element of size $n$:

$$\begin{aligned}&PS_{\mathfrak {g}}(f,g)(n) = V_{G_{\mathfrak {g}}}(f,g)(n+1) - V_{G_{\mathfrak {g}}}(f,g)(n) \\&\text {with}\ \ V_{G_{\mathfrak {g}}}(f,g) = [G_{\mathfrak {g}}(f,ng)_{n=\infty ..1}, AG_{\mathfrak {g}}(f,ng)_{n=0..\infty }];\nonumber \\&G_{\mathfrak {g}}(f,ng) = \frac{\mathrm{Vol}(\gamma _{\mathfrak {g}}(f,ng))}{\mathrm{Vol}(f)} ;\ \ \ \mathrm{Vol}(f) = \sum _x(f(x))\nonumber \\&AG_{\mathfrak {g}}(f,ng) = \frac{\mathrm{Vol}(\varphi _{\mathfrak {g}}(f,ng))}{\mathrm{Vol}(f)}; \nonumber \end{aligned}$$

(1)

The structuring element $g$ can be considered as a measure unit. Choosing a small structuring element allows to obtain a fine resolution in the texture description. Nevertheless, when information is required for larger object/distance, a small structuring element induces the requirement of a great iteration count and consequently a great computational cost.

Results present in Fig. 5 were obtained using a square structuring element of size $3 \times 3$. Some differences appear between the processed results and the expected ones. First difference comes from the volume normalization by the volume of the original image. This normalization prevents the magnitude variation of Dirac function depending on the contrast variation or the object count variation. Another difference appears for object count variation, the Dirac function in the object side presents the same magnitude in all the five cases. The volume normalization explain this behavior. Then, the last difference appears for the inter-distance variation, and in this case, the Dirac function magnitude increases with the inter-distance. Such behavior is due to the iteration scheme, more important is the number of iterations to fill the gap between object, more important are the obtained volume and consequently the Dirac magnitude. In this case, these two variations are linked.

2.3 Morphological covariance

Morphological covariance [7] characterizes the texture by analyzing the object appearance frequency in the image. Morphological covariance is computed from a function $\xi$ and a pair of points $P_2$ separated by a vector $\mathop {v}\limits ^{\rightarrow }$ (Eq. 2). The function $\xi$ commonly used is an erosion, but the opening was also used [8]. The main writing form is the normalized one (Eq. 3):

$$\begin{aligned}&K_{\mathfrak {g}}(f,P_{2,v}) = {\textit{Vol}}(\xi _{\mathfrak {g}}(f,P_{2,v}))\end{aligned}$$

(2)

$$\begin{aligned}&K_{\mathfrak {g}}^n(f,P_{2,v}) = \frac{{\textit{Vol}}(\xi _{\mathfrak {g}}(f,P_{2,v}))}{Vol(f)} \end{aligned}$$

(3)

The vector $\mathop {v}\limits ^{\rightarrow }$ can take different directions and different lengths, and then, $K(f,P_{2,v})$ is the concatenation of the different response among $\mathop {v}\limits ^{\rightarrow }$. For our example (Fig. 5), $\mathop {v}\limits ^{\rightarrow }$ has a length of 50 pixels and takes four directions: 0$^\circ$, 45$^\circ$, 90$^\circ$ and 135$^\circ$. The morphological covariance responses to images of squares (Fig. 2) are similar to the expected spectra (Fig. 4). These spectra show the symmetrical texture behavior for the horizontal, vertical and diagonal axes, and the pattern frequency.

2.4 Fractal signature

Fractal signature expression is derived from the fractal dimension calculation introduced by Mandelbrot [9]. It characterizes the complexity of the objects whose structure is invariant under scaling. The expression comes from the Minkowski–Bouligand dimension estimation.

The extension to image of the Minkowski–Bouligand dimension induces to estimate the volume evolution between two surfaces covering the image at different scales $i$. Both surfaces are called upper $U$ and lower $L$ and are numerically obtained by an erosion and a dilation (Eq. 5) using a non-flat structuring element $g$. Peleg proposed to use as non-flat structuring element a diamond shape $g_\diamond$ of size $3\times 3$ [10].

$$\begin{aligned}&\hbox {Sign}_i = \mathrm{Var}_{S_i} - \mathrm{Var}_{S_{i - 1}}\end{aligned}$$

(4)

$$\begin{aligned}&\text {with}\ \ \mathrm{Var}_{S_i} = S_i - S_{i - 1};\nonumber \\&S_i = \frac{\mathrm{Vol}_i - \mathrm{Vol}_{i - 1}}{2};\ \ \ \mathrm{Vol}_i = \sum _{x}(U_i(x) - L_i(x));\nonumber \\&\text {and}\ \ L_i(x) = \varepsilon _{\mathfrak {g}}(f,i g_\diamond )(x);\ \ U_i(x) = \delta _{\mathfrak {g}}(f,i g_\diamond )(x) \end{aligned}$$

(5)

To obtain a similar formulation to pattern spectrum signature, we use the inferior and superior signature as described by Peleg [10]. These signatures are obtained in the same way using the inferior volume $\mathrm{Vol}_\mathrm{inf}$ (Eq. 6) and superior volume $\mathrm{Vol}_\mathrm{sup}$ (Eq. 7).

$$\begin{aligned}&\mathrm{Vol}_{\mathrm{inf}_i} = \sum _{x}(f(x) - L_i(x))\end{aligned}$$

(6)

$$\begin{aligned}&\mathrm{Vol}_{\mathrm{sup}_i} = \sum _{x}(U_i(x) - f(x)) \end{aligned}$$

(7)

Results of fractal signature for the test images (Fig. 2) are close to the expected spectra. However, as for pattern spectra, some differences exist. The first one is for the inter-object variation and the object count variation. As for pattern spectrum, the magnitude of fractal signature increases on the background side due to the volume growth once the gaps filled during the closing process. Larger the inter-object distance is, higher the volume once the gaps filled is.

3 Color operators based on distance function

Computation of pattern spectrum, morphological covariance and fractal signature requires the definition of color erosion, dilation, opening and closing. Consequently, color ordering must be defined. In recent years, a lot of methods to define minimum ($\bigvee$) and maximum ($\bigwedge$) operations in color spaces are developed [11, 12]. As these approaches are not able to work with non-flat structuring element, dilation and erosion of an image $f$ by the structuring element $g$, in n-dimensional space, are then expressed as:

(8)

(9)

where $\mathcal {D}_f$ and $\mathcal {D}_g$ are, respectively, the spatial support of the image and the structuring element.

The aim of this paper is not to compare color mathematical morphology methods but to explain the interest of morphological tools. From conclusions of papers [1, 13], we select the “Convergent Color Mathematical Morphology” (CCMM) method. This method is based on the construction of an order according to the distance from reference color coordinates ($O^{-\infty }$ and $O^{+\infty }$). So, minimum and maximum between two colors, $C_1$ and $C_2$, could be define by:

$$\begin{aligned}&\bigwedge (C_1, C_2) \Leftrightarrow |\overrightarrow{C_1O^{-\infty }}| \le |\overrightarrow{C_2O^{-\infty }}|\end{aligned}$$

(10)

$$\begin{aligned}&\bigvee (C_1, C_2) \Leftrightarrow |\overrightarrow{C_1O^{+\infty }}| \le |\overrightarrow{C_2O^{+\infty }}| \end{aligned}$$

(11)

where $O^{-\infty }$ and $O^{+\infty }$ are, respectively, the convergence coordinates of the erosion and the dilation. In Eqs. (10) and (11), the vector norm $|.|$ uses the perceptual distance $\Delta E$ computed in CIELAB [14]. The complete method writing is in the thesis [15].

4 Preliminary results

To assess the discriminatory aspect of the color morphological texture attributes, we use the Outex database. This initial choice was made due to the classical use of this texture database for texture classification purpose. As we focus on texture discrimination, reduced to texture acquired with the same acquisition conditions, we selected the contest “Outex_TC_00013” [16]. This texture set contains 68 images sorted into 12 categories. For the same reasons, we constructed our classification test using the protocol proposed by Arvis [17]. Under this protocol and to take into account the reduced image count in some categories, an image is considered as a class and is divided into thumbnail of size $128\times 128$ pixels.

Always with the aim of better understanding the texture feature response to image content, we select three images with different texture contents (Fig. 6). In Figs. 7, 8 and 9, the feature signatures are provide for 20 thumbnails of the three images.

Fig. 6

Three used images of Outex database

Fig. 7

Superposition of 20 spectra of thumbnails for image

Fig. 8

Superposition of 20 spectra of thumbnails for image

Fig. 9

Superposition of 20 spectra of thumbnails for image

First ascertainment lies in the reduced standard deviation observed for pattern spectra and fractal signatures. This stability characterizes a low sensitivity to clarity variations, thanks to contrast-based processing, rather than on absolute values. In addition, pattern spectra and fractal signatures are clearly different for the three kinds of images.

For the first texture image (Fig. 6a), pattern spectrum and fractal signature (Fig. 7a, c), the first peak after the origin in the background zone characterizes the distance between vertical lines (interline distance around 20 pixels). Due to the symmetrical progression of the dilation, the gap between the vertical lines is filled after $d/2$ iteration, with $d$ the distance between the lines. In the same way, for the third images, pattern spectrum and fractal signature (Fig. 9a, c) highlight the same manner the circle frequency (around 30 pixels). High frequencies of Fig. 6b are characterized by a dominant peak for very low values located around the spectra origin (Fig. 8a, c).

Morphological covariance is adapted for oriented and periodic texture analysis. For the first image, morphological covariance allows to discriminate the texture orientation and the periodicity (Fig. 7b). For this image, the line periodicity is defined by the morphological covariance periodicity for angles $0, 45$ and $135^\circ$. Horizontal periodicity is detected for the first signature part (angle $=0^\circ$) and confirmed in the second and fourth ones. The third part ($90^\circ$), more perturbed, highlights canvas fibers on the vertical line.

In the case of Fig. 6c, patterns are larger than those present in the canvas images (Fig. 6a), then covariance spectra are composed of a single peak for the four directions. For images of fine texture random (Fig. 6b), the covariance presents low variations. In addition, in this case morphological covariance spectra have the disadvantage of important standard deviation due to the sensitivity to illumination change. This problem increases with the saturation of images (Fig. 8b).

The second test is based on a classical classification scheme, using the image set “Outex_TC_00013.” In order to be more generic as possible, we used the classification algorithm proposed by Arvis [17] using k-nearest neighbors. Improving the classification and learning steps are not in the topic of this paper. In Table 1, we present the good classification rates obtained by pattern spectrum, morphological covariance and fractal signature. The two first columns present the results using one and three nearest neighbors. For this texture image set, pattern spectrum is most discriminant, before the fractal signature and then the morphological covariance. As previously described for the pattern spectrum and fractal signature, the contrast-based processing is robust to lightning variations in the image. Consequently, results are stable when more than one neighbor is used in the classification.

Table 1 Correct classification percentage for different color texture features with the KNN method using 1 or 3 neighbors; using or not the color average

In Sect. 2, we explained that as the pattern spectrum and the fractal signature are based on contrast variations, they do not embed the grayscale average and by extension the color average. So the third column of Table 1 presents the classification results embedding the color average as proposed by Arvis. As expected, good classification rate is improved by this information addition; nevertheless, the pattern spectrum grows up with more than 10 % reaching 83 %, proving the potential of color multiscale texture features.

This classification highlights the importance of color information in color texture discrimination. Nevertheless, for OUTEX database results obtained are worse than those using more classical grayscale features [17, 18]. Recently, in [19], it was shown that the spatio-chromaticity of OUTEX images is weak, even if OUTEX database is presented as a color texture image database. Consequently, vector textures are not optimal in this case, and they introduced bias in the texture discrimination.

5 Conclusion

In this paper, we proposed a color and vector construction of morphological spectra. Using a perceptual distance function, the proposed texture features respected the actual knowledge about the human visual system. The proposed construction is fully generic due to the use of distance function and can be easily extended to spectral domain. In such case, selecting the right distance function opens the door to spectral texture features.

We show the interest and readability of these spectra for basic textures, and we investigated their discriminating aspect in a preliminary classification task. As the existing texture image database presents a low level of spatio-chromatic complexity, classical gray-level approaches are more efficient than vector ones. Nevertheless, the good classification rate obtained by pattern spectra is highest than 83 %, close to the results obtained by the gray-level approaches.

For the moment, no texture image database is really adapted for color texture classification, due to their reduced spatio-chromatic complexity. So it is not possible to assess performances of vector texture features. Due to this important requirements, an international work is in progress to specify and construct such image database. The construction of new texture features is processed in a vector way with the important question of the spatio-chromatic complexity. Following the construction of a new database most suitable, vector texture attributes can then be compared with other nonvector texture attributes [20].