Soft Color Morphology: A Fuzzy Approach for Multivariate Images

Abstract

Mathematical morphology is a framework composed by a set of well-known image processing techniques, widely used for binary and grayscale images, but less commonly used to process color or multivariate images. In this paper, we generalize fuzzy mathematical morphology to process multivariate images in such a way that overcomes the problem of defining an appropriate order among colors. We introduce the soft color erosion and the soft color dilation, which are the foundations of the rest of operators. Besides studying their theoretical properties, we analyze their behavior and compare them with the corresponding morphological operators from other frameworks that deal with color images. The soft color morphology outstands when handling images in the CIEL${}^*a{}^*b{}^*$ color space, where it guarantees that no colors with different chromatic values to the original ones are created. The soft color morphological operators prove to be easily customizable but also highly interpretable. Besides, they are fast operators and provide smooth outputs, more visually appealing than the crisp color transitions provided by other approaches.

1 Introduction

Mathematical morphology was introduced for binary images by Serra and Matheron [18]. It is based on two inexpensive operators, the erosion and the dilation, that offer a very interesting trade-off regarding their computational complexity and their expressive power. Morphological dilation makes objects bigger while morphological erosion shrinks them, but they are not each other’s inverse. They can be combined to design a myriad of image processing operators. The closing, defined as the erosion of the dilation, removes the small objects of the original image; the opening is the dual operator, filling the small gaps between objects. Other operators include the top-hat by closing and top-hat by opening, morphological template-matching [15], morphological filters and contrast mappings [1].

Mathematical morphology was promptly extended to grayscale images [19], but dealing with natural color images or with multivariate images proved to be more challenging. From the vast amount of approaches proposed in the literature, one can conclude that there is no successful definition for color mathematical morphology operators commonly accepted. We remark that the well-known lattice-based definition of erosion and dilation [26] cannot be applied straightforwardly to multivariate images: in contrast to binary or grayscale images, multivariate images do not have a consistent order. Different prior knowledge of the task at hand suggest different color orderings [30]. More formally, for any total order $\leqslant$ of a color space embedded in ${\mathbb {R}}^n$, it is possible to find three colors such that $c_1 \leqslant c_2 \leqslant c_3$, and such that the Euclidean distance between $c_1$ and $c_2$ is large but the Euclidean distance between $c_1$ and $c_3$ is small [11]. Thus, there is no total order between colors that is always consistent with human perception.

Fuzzy techniques have been used to extend binary morphological operators to grayscale ones [8, 12, 20]. They are employed with non-binary structuring elements in order to account for spatial uncertainty. Thus, one can design algorithms that are more robust and can handle a wide range of difficulties such as noise and artifacts. To the authors’ knowledge, there are no color morphological operators that generalize the fuzzy mathematical morphology ones. We aim at developing such a generalization to transfer the strengths of fuzzy mathematical morphology to multivariate image processing.

1.1 Related Literature

The vast majority of color morphology paradigms—which have been scarcely used by practitioners—can be divided into component-wise and vector approaches [2]. The former ones, such as the one presented by Gu et al. [17], consider a multivariate image as a stack of grayscale layers and process each of them independently. They do not exploit spectral correlations and are prone to introducing new colors.

The vector approach to color morphology, on the other hand, encompasses the methods that process all the channels of the image simultaneously. All of them order the colors somehow. Sometimes, they specify a fixed total order to then apply the dilation and erosion according to the lattice-based definitions of erosion and dilation [26], avoiding thus the creation of new colors. Aptoula and Lefèvre [3] use the lexicographic ordering (i.e., order by first component, resolve ties by second, and so on). Chanussot and Lambert [10] present a variation of the lexicographic ordering, in which they employ an 8-bit RGB representation of images. In this case, the higher bits of each channel are given more importance than the lower bits of the same or other channels. Although the behavior provided by this ordering is more aligned with human perception, it is arbitrary and not necessarily meaningful. For instance, the human eye can distinguish more green shades than red shades. Moreover, the channels of hyperspectral imagery may have different statistical distributions which makes them incomparable. Another variant of the lexicographic order is employed by Louverdis et al. [23], where the HSV color space is used, ordering the colors according by minimizing their value, then maximizing their saturation and finally minimizing their hue. Sartor and Weeks [24] present a method that orders colors based on their distance with respect to a reference color. This approach provides results that match the expectations, specially when the reference color has been carefully selected. Its drawbacks are the need of such user-supplied color (which could also be thought as a desirable customization) and the theoretical instability: two very distant colors may change their relative ranking due to small perturbations. Witte et al. [13] employed the fuzzy mathematical morphology operators with multivariate images by extending the usual operations. Specifically, they extend the notions of minimum, maximum, addition, negation and product for colors encoded in several color spaces—RGB, HSV and CIEL${}^*a{}^*b{}^*$. Accordingly, they consider structuring elements encoded in the same color spaces. Bouchet et al. [9] used a fuzzy order to create a total order between RGB triplets. For each channel, they learn a pixel-wise fuzzy preference relation. The three fuzzy preference relations are then aggregated with the arithmetic mean. Valle and Valente [27] propose a total order in the CIEL${}^*a{}^*b{}^*$ color space, based on the distance between colors and their relative position. They propose a binary operation between colors in the CIEL${}^*a{}^*b{}^*$ complete lattice that is associative, commutative and has an identity element. Employing it, they define morphological operators that admit structuring elements also encoded in the CIEL${}^*a{}^*b{}^*$ color space.

A series of works also explore adaptive orderings. All of them follow the same strategy than the previous algorithms—finding a total order between colors—but they do so in an image-dependent fashion. Velasco and Angulo [30] study and compare several supervised and unsupervised methods to reduce the dimensionality of the color space: principal component analysis, support vector machine, reference color and others. These approaches typically assume that small objects are located on the foreground, whereas large objects conform the background. One of such approaches by Velasco and Angulo [29] is the random projection technique: they employ a one-dimensional measure of the degree of centrality of a sample—known as a depth function—, and maximize it for each possible projection of the multivariate data into a one-dimensional space. Benavent et al. [5] also considered an adaptive ordering, which in this case is based on the image histogram. By considering a smoothed version of the histogram, they obtain a measure of how frequent a certain color is within the image. To decrease the computational complexity, they average the histograms of multiple images to obtain a reference color ordering for new, similar images. Lézoray [22] learns a color ordering from the image, being thus an adaptive ordering. It does so by learning a rank transformation on a complete lattice—equivalent to a total order. Laplacian eigenmaps are used to learn a nonlinear projection of the color space.

1.2 Goal of This Work

In this paper, we cover the definitions of the Soft Color Morphology operators—fuzzy, multivariate operators—and we study their properties. A preliminary version of these operators can be found in [7]. Besides, we compare them with the most interesting definitions of multivariate mathematical morphology that have been recently published in the literature. Such comparison is based on two points of view. First, their main theoretical properties are studied and summarized. Second, several visual examples are shown to provide insights into the behavior of each multivariate mathematical morphology alternative. These visual examples include different morphological operators applied to natural and artificial images.

The structure of the rest of the document is as follows. In Sect. 2, we introduce formal notations to handle colors and images. The basic operators of the soft color morphology are formally defined in Sect. 3, and their theoretical properties are studied in Sect. 4. Their behavior is visually presented and compared with other approaches in Sect. 5, along with some remarks regarding their usage. We discuss in depth such comparison in Sect. 6, that we conclude with the strengths and weaknesses of the soft color morphology operators.

2 Preliminaries

In this section, we introduce the concepts needed to define the soft color morphology operators.

2.1 Formal Description of Images

First, we formally describe the objects we are dealing with, such as colors, channels and images.

Definition 1

We define the following elements.

• A channel, ${\mathscr {C}}$, is a set of real values, ${\mathscr {C}}\subset {\mathbb {R}}$.

• A color space, ${\mathfrak {C}}$, is the cartesian product of a series of channels ${\mathfrak {C}}= {\mathscr {C}}_1 \times \cdots \times {\mathscr {C}}_m$.

• A ${\mathfrak {C}}$-encoded color is one element of the color space, $c \in {\mathfrak {C}}$.

• A ${\mathfrak {C}}$-encoded multivariate image or simply a ${\mathfrak {C}}$-encoded image, A, is a map $A: {\mathbb {Z}}^n \rightarrow {\mathfrak {C}}$, where n is its spatial dimension and ${\mathfrak {C}}$ is its color space.

• Any image A is associated with its support$d_A \subset {\mathbb {Z}}^n$, the region where the image is defined. We can consider A(x) to be meaningless for x outside the support $d_A$.

Some remarks must be mentioned. First, without loss of generalization, we will always consider that the first channel is ${\mathscr {C}}_1 = [0, 1]$. Otherwise, since ${\mathscr {C}}_1$ is a subset of ${\mathbb {R}}$, we can map it to [0, 1] with a monotonic bijection—either with a linear function if ${\mathscr {C}}_1 = [a, b]$, or with the sigmoid transformation for a general ${\mathscr {C}}_1 \subset {\mathbb {R}}$. Second, we emphasize that we consider images with generic spatial dimension n—which includes, for instance, volumetric 3D imagery. Although we focus on $n=2$ throughout this paper, our operators from Definitions 7 and 8 can process images having a generic spatial dimension n. Third, a grayscale image is a particular case of multivariate image, which has only one channel (that is, $m = 1$). Finally, the support of an image A is simply a region within the spatial location ${\mathbb {Z}}^n$ that indicates which part of the image is of interest. It can be assumed to contain no information outside it (e.g., initialized to a meaningless fixed value).

To compute our soft color morphology operators, we employ structuring elements whose values range in [0, 1]. We also define operators to deal with movements in the spatial domain.

Definition 2

We define the following elements.

• A structuring element, B, is a grayscale image $B: {\mathbb {Z}}^n \rightarrow [0, 1]$, where n is its spatial dimension.

• The reflection of a structuring element B is the structuring element $\overline{B}$ defined by $\overline{B}(x) = B(-x)$.

• The spatial translation by a vector $v \in {\mathbb {Z}}^n$, $T_v$, is a map from subsets of ${\mathbb {Z}}^n$ to subsets of ${\mathbb {Z}}^n$ such that

  $$\begin{aligned} \forall d\subset {\mathbb {Z}}^n, \forall a\in {\mathbb {Z}}^n, \quad a \in T_v(d) \iff a-v \in d. \end{aligned}$$

Similarly to multivariate images, a structuring element B is always associated with its support$d_B \subset {\mathbb {Z}}^n$. Outside their support, we will consider that structuring elements have a meaningless value (i.e., for all $x \not \in d_B$, $B(x) = 0$). We will always consider structuring elements with the same spatial dimension n than the images we are processing and that have a nonzero value in their domain.

2.2 The CIEL${}^*a{}^*b{}^*$ Color Space

To deal with images in the visible spectrum, we will consider the CIEL${}^*a{}^*b{}^*$ color space, which has three channels. The first one, L*, matches the human perception of lightness, whereas the other two, a* and b*, conceil its chromatic information. Besides, it is perceptually uniform: the Euclidean distance between two colors is proportional to their difference when perceived by humans [31]. For more information regarding the CIEL${}^*a{}^*b{}^*$ color space and its relation to other color spaces such as RGB and XYZ, see Wyszecki and Stiles [31].

Typical implementations of the CIEL${}^*a{}^*b{}^*$ color space consider the first channel to be ${L^{*}} = [0, 100]$. Throughout this paper, we divide such lightness values by 100 so that our first channel is exactly [0, 1].

Conversion between CIEL${}^*a{}^*b{}^*$ and grayscale is straightforward, as shown in the following definition.

Definition 3

(Conversion of CIEL${}^*a{}^*b{}^*$  to and from grayscale) The map $\pi$, that converts a CIEL${}^*a{}^*b{}^*$-encoded color $(A_1, A_2, A_3)$ into a grayscale color, is defined as:

$$\begin{aligned} \begin{array}{cccl} \pi : &{} {L^{*}} \times {a^{*}} \times {b^{*}} &{} \longrightarrow &{} [0, 1] \\ &{} \big ( A_1, A_2, A_3 \big ) &{} \longmapsto &{} A_1. \end{array} \end{aligned}$$

The map $\imath$, that converts a grayscale color G into a CIEL${}^*a{}^*b{}^*$-encoded color, is defined as:

$$\begin{aligned} \begin{array}{cccl} \imath : &{} [0, 1] &{} \longrightarrow &{} {L^{*}} \times {a^{*}} \times {b^{*}} \\ &{} G &{} \longmapsto &{} \big ( G, 0, 0 \big ). \end{array} \end{aligned}$$

That is, simply by reducing the CIEL${}^*a{}^*b{}^*$-encoded image into the L* channel (i.e., forgetting the channels a* and b*) provides a grayscale version.

2.3 Fuzzy Logic Operators

The soft color dilation and erosion are designed using fuzzy logic operators. In particular, we are interested in conjunctions and fuzzy implication functions.

Definition 4

A conjunctionC is a map $C:[0, 1] \times [0,1] \rightarrow [0, 1]$ that is increasing in both variables and that satisfies $C(1, 0) = C(0, 1) = 0$ and $C(1, 1) = 1$.

A conjunction C is a semi-norm if it satisfies the following border condition: for all $x \in [0, 1]$, $C(1,x) = C(x,1) = x$.

Definition 5

A fuzzy implication function is a map $I:[0, 1] \times [0,1] \rightarrow [0, 1]$ that is decreasing in the first variable and increasing in the second and that satisfies $I(1, 0) = 0$ and $I(0, 0) = I(1, 1) = 1$.

We say that a fuzzy implication function I fulfills the left neutrality principle (NP) if and only if $I(1, y) = y$ for all $y \in [0, 1]$, see [4].

Besides, for any conjunction C we define its residual operator as

$$\begin{aligned} I_C(x,y)= & {} \sup {\{t \in [0,1] \; | \; C(x,t) \leqslant y\}},\\&\text {for all } x, y \in [0, 1]. \end{aligned}$$

If $C(1,x) > 0$ for all $x > 0$, then $I_C$ is a fuzzy implication function and it is called the residual implication (or R-implication) of C. All R-implications fulfill (NP) [4].

Due to their use in the paper, special mention deserve the so-called triangular norms (t-norms, for short) [21], which are an important family of conjunctions, and their residual implications. The fuzzy logic operators used in the paper are summarized in Table 1. They are the pairs of t-norms and their corresponding R-implications: the minimum t-norm $T_\mathbf{M }$ and the Gödel implication $I_\mathbf{GD }$, the product t-norm $T_\mathbf{P }$ and the Goguen implication $I_\mathbf{GG }$, and the Łukasiewicz t-norm $T_\mathbf{LK }$ jointly with the Łukasiewicz implication $I_\mathbf{LK }$.

Table 1 Fuzzy operators used in this article

3 Soft Color Morphology

We now formally introduce the Soft Color Morphology basic operators. To do so, we extend the fuzzy mathematical morphology approach to the first channel, maintaining (and spreading) the values of the rest of the channels accordingly.

We define here a method to totally order the colors of an image in a specific neighborhood, which is used to define a ${{\mathrm{\mathbf {max}}}}$ and ${{\mathrm{\mathbf {min}}}}$ of a set of pixels. Let us assume, we are processing an image I at the location y, and that we are comparing some colors I(x) for x in the neighborhood of y. We are interested in the information contained in the first channel, and so we first order the spatial locations x by the first component of their color, $I_1(x)$. To deal with ties, which are uncommon in natural images, we first select the spatial location x closest—with regard to the Euclidean distance—to the location of pixel we are processing, y. Further ties are resolved with the lexicographic order. That is, ordering by the first channel, then by the second, and so on. The following definition formally introduces these ${{\mathrm{\mathbf {max}}}}$ and ${{\mathrm{\mathbf {min}}}}$ operators.

Definition 6

Let $I:{\mathbb {Z}}^n \rightarrow {\mathfrak {C}}$ be a multivariate image. Let us consider a location $y \in {\mathbb {Z}}^n$ and a neighborhood $N \subset {\mathbb {Z}}^n$. Then, the maximum and minimum of the colors I(x) with respect to y and N, $\textstyle \underset{x \in N}{\mathbf {max}^{y}}(I)$ and $\textstyle \underset{x \in N}{\mathbf {min}^{y}}(I)$, are defined as

$$\begin{aligned}&\left\{ \begin{array}{l} \nabla _1 =\underset{x \in N}{{\text {arg max}\,}}I_1(x), \\ \nabla _2 =\underset{x \in \nabla _1}{{\text {arg min}\,}}{d_2(x, y)},\\ \textstyle \underset{x \in N}{\mathbf {max}^{y}}(I) = \underset{x \in \nabla _2}{\max }^\text {lexic}{I(x)}, \end{array} \right. \\&\left\{ \begin{array}{l} \varDelta _1 =\underset{x \in N}{{\text {arg min}\,}}I_1(x), \\ \varDelta _2 =\underset{x \in \varDelta _1}{{\text {arg min}\,}}{d_2(x, y)},\\ \textstyle \underset{x \in N}{\mathbf {min}^{y}}(I) = \underset{x \in \varDelta _2}{\min }^\text {lexic}{I(x)}, \end{array} \right. \end{aligned}$$

where $d_2:{\mathbb {R}}^2 \rightarrow {\mathbb {R}}$ is the Euclidean distance and ${\max }^\text {lexic}$ and ${\min }^\text {lexic}$ are the lexicographic maximum and minimum.

We emphasize that ordering where ties may be resolved according to the spatial location is not appropriate to define idempotent opening and closing operations. However, any other ordering would introduce some bias toward a particular color—and we consider that there is not a consistent order among colors. Besides, such an ordering is needed to obtain the full color preservation property (see Proposition 2).

Once we have defined an order between colors, we can proceed to define the soft color dilation and soft color erosion. They are not defined using the lattice-based definition of morphological operators, but extending the fuzzy mathematical morphology definition to multivariate images.

Fig. 1

From left to right, soft color erosion (left), the L*a*b*-encoded Balloons image (center) and soft color dilation (right), with the minimum t-norm, its residuated implication (the Gödel implication) and a $31\times 31$-pixel Gaussian-shaped structuring element, with $\sigma = 8$ pixels and a maximum value of 1. We remark that the irregular shapes of the eroded balloons reflect their irregular illumination (Color figure online)

Definition 7

Let C be a conjunction, $A: {\mathbb {Z}}^n \rightarrow {\mathfrak {C}}$ a multivariate image and $B: {\mathbb {Z}}^n \rightarrow [0, 1]$ a structuring element. Then, the soft color dilation of A by B, ${\mathcal {D}}_C(A,B)$, is

$$\begin{aligned}&\displaystyle {\mathcal {D}}_C(A,B)(y) \\&\quad = \textstyle \underset{x \in d_A \cap T_y(d_B)}{\mathbf {max}^{y}} \bigg \{ \Big (C\big (B(x-y), A_1(x)\big ), \, A_2(x), \ldots , A_m(x) \Big ) \bigg \}. \end{aligned}$$

Essentially, in the absence of ties, the definition selects the spatial location x at which $C(B(x-y), A_1(x))$ is maximum (of course, restricted to the appropriate domain, $x \in d_A \cap T_y(d_B)$). Once the location is selected, the other channels are simply dragged.

Similarly, we define the soft color erosion as follows.

Definition 8

Let I be a fuzzy implication function, $A: {\mathbb {Z}}^n \rightarrow {\mathfrak {C}}$ a multivariate image and $B: {\mathbb {Z}}^n \rightarrow [0, 1]$ a structuring element. Then, the soft color erosion of A by B, ${\mathcal {E}}_I(A,B)$, is

$$\begin{aligned}&{\mathcal {E}}_I(A,B)(y) \\&\quad = \textstyle \underset{x \in d_A \cap T_y(d_B)}{\mathbf {min}^{y}} \bigg \{ \Big (I\big (B(x-y), A_1(x)\big ), \, A_2(x), \ldots , A_m(x) \Big ) \bigg \}. \end{aligned}$$

The soft color morphology operators interpret the first channel as a measure to indicate with which degree the pixels belong to the foreground or to the background. When processing natural images with the CIEL${}^*a{}^*b{}^*$ color space, the first channel is already very informative, containing the luminance of a color pixel and disregarding the chromatic information. Using such channel to discriminate whether the objects belong to the foreground or the background provides a generalization of the same procedure for grayscale and binary images, which is coherent with human visual perception. In specific applications, practitioners must necessarily provide this information in accordance with the task they are dealing with, since images from different problems can be of a really different nature. Besides the luminance channel L* in natural images, other examples are the green channel, in RGB-encoded retinal angiographies; or an artificial channel that orders the colors according to their frequency, such as in Benavent et al. [5]. Our operators are designed to easily handle these and similar cases.

Given these two definitions, other morphological operators are defined straightforwardly:

Definition 9

Let A be a multivariate image and let B be a structuring element. Let C be a conjunction and let I be a fuzzy implication function. Then, the closing of A by B, ${\mathcal {C}}_{C,I}(A,B)$, and the opening of A by B, ${\mathcal {O}}_{C,I}(A,B)$, are defined as:

$$\begin{aligned} {\mathcal {C}}_{C,I}(A,B) = {\mathcal {E}}_I\left( {\mathcal {D}}_C(A,B), \overline{B}\right) , \\ {\mathcal {O}}_{C,I}(A,B) = {\mathcal {D}}_C\left( {\mathcal {E}}_I(A,B), \overline{B}\right) . \end{aligned}$$

Figure 1 shows the soft color dilation and soft color erosion with the balloons image. As shown in this first example, the soft color dilation makes bright objects larger, while the soft color erosion diminishes them. We emphasize the behavior when two objects clash: either one object is placed “over” the other or they collide into a well-defined border.

In Fig. 2, the soft color opening and soft color closing operators are visualized processing a $600 \times 600$-pixel dermoscopic image in the CIEL${}^*a{}^*b{}^*$ color space. In it, the skin lesion has a globular network pattern. The opening provides a darker image, in which small bright blobs disappear, but the big ones remain. On the other hand, the closing operator has the same behavior but on the dark blobs, providing thus a brighter image.

Fig. 2

Soft color opening (left) and soft color closing (right) of a dermoscopy image (center) depicting a globular network, using the same configuration as in Fig. 1 (Color figure online)

Fig. 3

Soft color erosion (left column) and soft color dilation (right column) of the L*a*b*-encoded, $600\times 400$ image (center), with the minimum t-norm (top row), the product t-norm (middle), the Łukasiewicz t-norm (bottom) and their respective R-implications. It was used a $31\times 31$-pixel Gaussian-shaped structuring element ($\sigma = 8$ pixels and a maximum value of 1) (Color figure online)

In Fig. 3, the soft color dilation and soft color erosion operators are visualized processing a $600 \times 400$-pixel image in the CIEL${}^*a{}^*b{}^*$ color space. Different t-norms were used to visualize their different behavior: the closer a t-norm is to the minimum operator, the greater the impact in the final result. The minimum t-norm and the Gödel implication provide a stronger response, whereas the Łukasiewicz pair provides the softer one. This is specially noticeable in high-contrasted regions, such as the grapes, and in regions that present texture, such as the pineapple. Although we employed pairs of t-norm and R-implication, this does not need to be always the case. More visual examples are found in Sect. 5.

In Fig. 4, different structuring elements are used along with the minimum t-norm and the Gödel implication. In this $100\times 100$-pixel patch of the jellybeans image, we can visualize how different structuring elements affect the erosion and dilation operations. Whereas the $1\times 1$ impulsive structuring element leaves the image unaffected (both for the erosion and dilation operators), crisp structuring elements highlight limits between objects and their shades. On the other hand, non-binary structuring elements show a smoother response in such limits and can be tailored to be isotropic more easily.

Fig. 4

Soft color erosion (left) and soft color dilation (right) for several structuring elements (center): $1\times 1$ impulsive; $7\times 7$ squared flat; $7\times 7$ diamond-shaped flat; $9\times 9$ Gaussian-shaped; $13\times 13$ bar-shaped with decay; and $35\times 35$ right-neighborhood structuring element. We remark that the results of the impulsive structuring element are equal to the original image (Color figure online)

In Sect. 5, these operators are shown and are also compared with the corresponding operators derived from other morphological frameworks.

4 Properties

In this section, we study the properties of the soft color morphology operators from a mathematical point of view.

4.1 Chromatic Preservation in CIEL${}^*a{}^*b{}^*$

When dealing with natural images, it is natural to think that the erosion and dilation should not introduce colors whose shade or chromatic value was not originally present. The soft color morphological operators from Definitions 7 and 8 have been designed with this in mind, and so they preserve the chromatic information of colors when the CIEL${}^*a{}^*b{}^*$ color space is used.

Proposition 1

(Chromatic preservation in CIEL${}^*a{}^*b{}^*$) Let A be a CIEL${}^*a{}^*b{}^*$-encoded image with spatial dimension n, let C be a conjunction and I a fuzzy implication function, and let B be a structuring element.

Then, for all $y \in d_A$, there exist $x_1, x_2 \in d_A$ such that the components a* and b* of ${\mathcal {D}}_C(A,B)(y)$ and ${\mathcal {E}}_I(A,B)(y)$ are equal to, respectively, the components a* and b* of $A(x_1)$ and $A(x_2)$.

Proof

Recalling their definition, for a CIEL${}^*a{}^*b{}^*$-encoded image,

$$\begin{aligned}&\displaystyle {\mathcal {D}}_C(A,B)(y) \\&\quad = \textstyle \underset{x \in d_A \cap T_y(d_B)}{\mathbf {max}^{y}} \bigg \{ \Big (C\big (B(x-y), L^*(x)\big ), \, a^*(x), b^*(x) \Big ) \bigg \} . \end{aligned}$$

Thus, it is clear that the only channel that may have been modified is the first one: the combination $(a^*(x), b^*(x))$ previously existed (as a matter of fact, in the spatial location x). Since the chromatic information of a CIEL${}^*a{}^*b{}^*$-encoded color is determined by the combination of the second and third components, a* and b*, and these channels are preserved, we conclude that such chromatic information is indeed kept.

The same argument holds for the erosion. $\square$

Of course, this property also holds for the closing and the opening operators, since they are combinations of dilations and erosions.

4.2 Full Color Preservation

In some settings, we should avoid creating colors—not only the chromatic components, but all the components—that were not present in the original image. The creation of new colors, also called the false color problem, is sometimes described as a problem to be overcomed, although it may not necessarily be an undesired situation [28]. For instance, the fuzzy mathematical morphology, which we extend, comprises operators that may introduce new colors. However, the false color problem is more critical for multivariate images, due to, among other factors, the correlation between channels. Regarding color images, van de Gronde and Roerdink [28] consider operators invariant to hue rotations. Dealing with more general multivariate imagery, such as a corregistered PET and CT scan, may still make full color preservation a desirable property.

Our operators from Definitions 7 and 8 do preserve colors under certain conditions. Specifically, whenever we are using a binary structuring element and either a semi-norm (for the dilation) or a fuzzy implication function that fulfills (NP) (for the erosion). The following proposition provides sufficient conditions to enforce color preservation.

Proposition 2

(Full color preservation) Let A be a multivariate image with spatial dimension n, let C be a semi-norm, let I be a fuzzy implication function that fulfills (NP), and let B be a structuring element.

If $B(x) \in \{0, 1\}$ for all $x \in d_B$ and $B(0) = 1$, then ${\mathcal {D}}_C(A,B)$ and ${\mathcal {E}}_I(A,B)$ only contain colors that appear in A.

Proof

Let us see the case of the dilation. Since C is a semi-norm, $C(a, 1) = a$ and $C(0, a)= 0$ due to $C(0, a) \leqslant C(0, 1) = 0$ for all $a \in [0, 1]$. Thus, $C\big (B(x-y), A_1(x)\big )$ is either $A_1(x)$ or 0. If any of such values is different to zero, then the dilation in the location $y \in {\mathbb {Z}}^n$ is the maximum among those colors, all of which appear in A. Otherwise, if all values are 0, since $B(0) = 1$ we can derive that $A_1(y) = 0$ and, thus, ${\mathcal {D}}_C(A,B)(y) = A(y)$ due to the choice procedure in case of ties.

The erosion has a similar proof. All fuzzy implication functions are increasing in the second variable, so $I(0, a) = 1$ for all $a \in [0, 1]$ since $I(0, a) \geqslant I(0, 0) = 1$ for all $a \in [0, 1]$. Since the fuzzy implication function fulfills the left neutrality principle (NP), $I(1, a) = a$ for all $a \in [0, 1]$. Thus, $I\big (B(x-y), A_1(x)\big )$ is either $A_1(x)$ or 1. The same argument that in the dilation proof concludes that the color provided was already present in the original image A. $\square$

We remark that the opening and the closing do not create new colors. Provided the hypotheses of the proposition are satisfied, the dilation and the erosion do not create new colors and, thus, so cannot combinations of them.

4.3 Generalization of the Fuzzy Mathematical Morphology Operators

Another interesting property of these operators is that they constitute a generalization of the fuzzy mathematical morphology ones [8] when applied to CIEL${}^*a{}^*b{}^*$-encoded images.

Proposition 3

(Generalization of fuzzy mathematical morphology for grayscale images) Let us consider the conversion between CIEL${}^*a{}^*b{}^*$ and grayscale colors from Definition 3. Then, the soft color morphology operators, when applied to grayscale images encoded in CIEL${}^*a{}^*b{}^*$, coincide with the corresponding operators from the fuzzy mathematical morphology paradigm using the same structuring element.

Proof

To see this, we must prove that, when restricted to grayscale images, our new operators provide exactly the same results.

We begin by considering the dilation. Let G be a grayscale image, B be a structuring element and C be a conjunction. Then, the CIEL${}^*a{}^*b{}^*$ conversion of G is $\imath (G)$, where $\imath (G)(x) = (G(x), 0, 0)$, and its support remains unchanged, $d_{\imath (G)} = d_G$.

The dilation of $\imath (G)$ by B is then

$$\begin{aligned} \begin{array}{lcl} {\mathcal {D}}_C(\imath (G), B)(y) &{} = \bigg \{ &{} \Big (C\big (B(x-y), G(x)\big ), \, 0, \, 0 \Big ) \text { st. } \\ &{}&{} x \in d_G \cap T_y(d_B) \text { and }\\ &{}&{} C\big (B(x-y), G(x)\big ) \text { is maximum} \bigg \}. \end{array} \end{aligned}$$

Regardless the result of the choice procedure in case of ties, the grayscale projection of such image is:

$$\begin{aligned} \begin{array}{ll} \pi \big ( {\mathcal {D}}_C(\imath (G),B)(y) \big ) &{} = \bigg \{ C\big (B(x-y), G(x)\big ) \text { st. } \\ &{} \quad x \in d_G \cap T_y(d_B) \text { and } C\\ &{}\quad \big (B(x-y), G(x)\big ) \text { is maximum} \bigg \} \\ &{} = \underset{x \in d_G \cap T_y(d_B)}{\text {max}}\bigg \{ \Big (C\big (B(x-y), G(x)\big ) \bigg \}, \end{array} \end{aligned}$$

which matches the definition of dilation from the fuzzy mathematical morphology paradigm. The case for erosion is proved in a similar way and holds for any structuring element B and fuzzy implication function I. $\square$

4.4 First Channel Monotonicity

The aforementioned generalization of the fuzzy mathematical morphology is an interesting property because it can be leveraged to prove several properties, such as the monotonicity of the soft color morphological operators. To argue about monotonicity, we must be able to compare both ${\mathfrak {C}}$-encoded colors and ${\mathfrak {C}}$-encoded images.

This motivates the following definition of h-order [16]:

Definition 10

Given a complete lattice ${\mathcal {L}}$ and a surjective mapping $h:{\mathfrak {C}}\rightarrow {\mathcal {L}}$, the h-order is the relation ${\,\leqslant _{h}\,}$ defined as:

$$\begin{aligned} c_1 {\,\leqslant _{h}\,}c_2 \iff h(c_1) \leqslant h(c_2), \quad \forall c_1, c_2 \in {\mathfrak {C}}. \end{aligned}$$

We order the colors according to the first component. In particular, when considering the CIEL${}^*a{}^*b{}^*$ as the color space to be used, the colors are ordered by their luminance. To do so, we consider the projection on the first component, $\pi :$CIEL${}^*a{}^*b{}^*$$\rightarrow [0, 1]$ such that $\pi (l, a, b) = l$. Thus, we consider its h-order, obtaining the following definition:

Definition 11

(Partial order in CIEL${}^*a{}^*b{}^*$) Given two CIEL${}^*a{}^*b{}^*$ colors, $c_1 = (l_1, a_1, b_1)$ and $c_2 = (l_2, a_2, b_2)$, we say that $c_1$is lighter or equal than$c_2$, denoted by $c_1 {\,\leqslant _\pi \,}c_2$ if and only if $l_1 \leqslant l_2$.

Besides, given two CIEL${}^*a{}^*b{}^*$-encoded images with the same spatial dimension and the same domain, U and V, we say that U is (pixel-wise) lighter or equal than V, denoted as $U {\,\subseteq _\pi \,}V$ if for all $x \in d_{U} = d_{V}$, it holds that $U(x) {\,\leqslant _\pi \,}V(x)$.

We remark that, although we introduce now an h-order, our mathematical morphology operators are not based on an h-order, but on the total order presented in Definition 6. Although such order resembles an h-order, it additionally relies on the spatial location of pixels to resolve ties. This h-order is introduced in relation with the first channel monotonicity and the first channel adjunction.

Proposition 4

The relations from Definition 11 are, respectively, a preorder (i.e., a reflexive and transitive relation) for CIEL${}^*a{}^*b{}^*$-encoded colors, and a preorder for CIEL${}^*a{}^*b{}^*$-encoded images.

We remark that such relations are not partial orders, since antisymmetry does not hold. We can now state the monotonicity of our operators as follows:

Proposition 5

(Monotonicity) Let A be a CIEL${}^*a{}^*b{}^*$-encoded image and B a structuring element such that $B(0) = 1$. Let C be a conjunction and I a fuzzy implication function. Then,

$$\begin{aligned} {\mathcal {E}}_I(A,B) {\,\subseteq _\pi \,}A {\,\subseteq _\pi \,}{\mathcal {D}}_C(A,B). \end{aligned}$$

(1)

Besides, if I is the R-implication of C, then,

$$\begin{aligned}&{\mathcal {E}}_I(A,B) \nonumber \\&\quad {\,\subseteq _\pi \,}{\mathcal {O}}_{C,I}(A,B) {\,\subseteq _\pi \,}A {\,\subseteq _\pi \,}{\mathcal {C}}_{C,I}(A,B) {\,\subseteq _\pi \,}{\mathcal {D}}_C(A,B). \end{aligned}$$

(2)

Proof

Since our operators generalize the ones of the fuzzy mathematical morphology (Proposition 3) and the order only takes into account the first component (Definition 11), we can translate the result in [20, Proposition 45] to our framework. $\square$

4.5 Adjunction

The fuzzy mathematical morphology, which we extend, has the property of being an adjunction [20, Proposition 52]: under certain constraints on the conjunction C and the fuzzy implication function I, $A_1 \subseteq {\mathcal {E}}_I(A_2, B)$ if and only if ${\mathcal {E}}_C(A_1, \overline{B}) \subseteq A_2$. We remark that we use the reflection of the structuring element, $\overline{B}$, given our specific definition of erosion and dilation, similar to the definition by Kerre et al. [20, Definition 12].

Although we consider that the inclusion is not well defined between multivariate images, we can transfer this property to our settings if we restrict the inclusion to the first channel as an h-adjunction [16]. Thus, considering the first channel inclusion above defined, ${\,\subseteq _\pi \,}$, we may state the following property.

Proposition 6

(h-Adjunction) Let C be a left-continuous t-norm and I its residuated implication and B a structuring element. Then, the operators ${\mathcal {E}}_I(\cdot , B)$ and ${\mathcal {D}}_C(\cdot , \overline{B})$ form a $\pi$-adjunction. That is, for all images $A_1, A_2$,

$$\begin{aligned} A_1 {\,\subseteq _\pi \,}{\mathcal {E}}_I(A_2, B) \iff {\mathcal {D}}_C(A_1, \overline{B}) {\,\subseteq _\pi \,}A_2. \end{aligned}$$

Proof

Again we use the fact that our operators generalize the ones of the fuzzy mathematical morphology (Proposition 3) and the order restricted to the first component matches that of the fuzzy mathematical morphology. Therefore, we can translate the result in [20, Proposition 52] to our framework. $\square$

5 Comparison Between Color Morphologies

In this section, the Soft Color Morphology operators are compared with other morphological operators that deal with color images. They are compared both in terms of their characteristics and with visual examples. The former are collected in Table 2, which provides practical information about the basics of each algorithm: whether they preserve colors, their computational complexity, and so on. The latter are meant to provide an insight into their behavior and compare it with the other frameworks. Since we are designing general operators, our aim is to make them as interpretable as possible, so practitioners know when to apply them. We remark that a quantitative comparison is not possible: performance metrics can only be used when facing a specific application, which is out of the scope of this paper.

Table 2 Comparison of different approaches to color mathematical morphology

For the rest of the section, we consider all the morphological frameworks contained in Table 2. Among other features, it explains the type of order between colors: marginal, for channel-wise processing; total, when all colors have a fixed ranking. The approach by Witte et al. [13] does not need to consider a color ordering to process them. The complexity of all of them is at least as high as computing the lattice-based definition of morphological operators, but it is higher in some situations: the histogram-based version by Benavent et al. [5] requires computing a histogram of size $100\times 100 \times 100$ and filter it (or precomputing it to process images that are similar to a certain database); the random projections [29] technique requires approximating a supremum over an infinite set of projections with a maximum over a finite random projections uniformly distributed (specifically, using 1000 projections in the original paper).

The color preservation, chromatic preservation in natural images and generalization of more elementary mathematical morphology frameworks are interpreted as in Propositions 1, 2 and 3. Although all works extend mathematical morphology, not all of them generalize more elementary operators. Specifically, we understand it as the fact that performing some elementary operations on any elementary image coincides with mapping it into the multivariate space, performing some operations and projecting it back into the elementary space. In particular, the operators that behave differently based on the image to be processed lose the ability to distinguish black from white, and as a consequence their respective dilations may make bright objects smaller (if they consider it to be background, for instance).

We emphasize that our method can consider structuring elements being grayscale images, whereas other approaches, [13] and [27], consider multivariate structuring elements. Lastly, the customization classification goes beyond changing the structuring element: the color reference in the CIEL${}^*a{}^*b{}^*$ space [24] and in the CIEL${}^*a{}^*b{}^*$ quantale [27] depend on a user-chosen color; the histogram-based morphology [5] admits coarser or finer smoothings of the histogram, and the soft color morphology depends on a fuzzy conjunction and fuzzy implication function that model its behavior for non-binary structuring elements.

Fig. 5

Erosion by different morphologies, being (1) original $512 \times 512$ image, (2) $7\times 7$ binary structuring element, a component-wise [17], b bit mixing [10], c color reference for white [24], d lexicographic order for VSH [23], e multivariate fuzzy operations [13], f histogram-based [5], g random projections [29], h color reference in CIELab Quantale [27], i soft color morphology (Color figure online)

Fig. 6

Dilation by different morphologies, being (1) original $512 \times 512$ image, (2) $13\times 13$ bar-like structuring element, (3) binarization of (2), a component-wise [17], b bit mixing [10], c color reference for white [24], d lexicographic order for VSH [23], e multivariate fuzzy operations [13], f histogram-based [5], g random projections [29], h color reference in CIELab Quantale [27], i soft color morphology (Color figure online)

Fig. 7

Opening by different morphologies, being (1) original image, (2) $50 \times 50$ detail of the original image, (3) $9 \times 9$ Gaussian structuring element, (4) binarization of (3), a component-wise [17], b bit mixing [10], c color reference for white [24], d lexicographic order for VSH [23], e multivariate fuzzy operations [13], f histogram-based [5], g random projections [29], h color reference in CIELab Quantale [27], i soft color morphology (Color figure online)

Fig. 8

Closing by different morphologies, being (1) original $611 \times 764$ image, (2) $7 \times 7$ binary structuring element, a component-wise [17], b bit mixing [10], c color reference for white [24], d lexicographic order for VSH [23], e multivariate fuzzy operations [13], f histogram-based [5], g random projections [29], h color reference in CIELab Quantale [27], i soft color morphology (Color figure online)

Figures 5, 6, 7 and 8 shows the representations of the same morphological operator for the different methods in Table 2. All methods have been executed with the values of the parameters recommended in the original articles, using white reference for the method by Sartor [24] and black as the color reference on the CIEL${}^*a{}^*b{}^*$ quantale [27]. Specifically, the method based on the multivariate fuzzy operators by Witte et al. is executed with the CIEL${}^*a{}^*b{}^*$ color space, the minimum t-norm and its residuated R-implication [13]. The soft color morphology method also employs the minimum t-norm as conjunction and its R-implication, the Gödel implication. To compare a method that admits non-binary structuring elements with another one limited to binary structuring elements, whenever a non-binary structuring element is considered, it is thresholded at the level 0.5 to be used by the latter. Conversely, grayscale structuring elements are mapped into the CIEL${}^*a{}^*b{}^*$, according to Definition 3 for the method by Witte et al. [13], and based on the unity element for the method by Valle et al. [27]. Lastly, all methods that employed an adaptive ordering, did so the first time the original image was processed (not after each erosion or dilation).

Figures 5 and 6 depict the basic operators: erosion and dilation. We remind that erosion is meant to shrink objects, whereas dilation enlarges them. Color reference for white seems to work in an opposite direction as the other frameworks, whereas random projections, being image dependent, works counterintuitively in the first case, but intuitively in the second. It is noticeable how the histogram-based technique provides a sharp result in Fig. 5 and blurs the original image in Fig. 6, to the point of removing small objects, due to its image-dependent behavior. The method based on lexicographic order for the channels VSH provides a singular result: the erosion contains large bright areas, such as the nose. The method based on extending fuzzy operators to multivariate data appears similar to the Soft Color Morphology in Fig. 5 but is clearly different in Fig. 6. This seems to be caused by how the different operations are generalized. Although the result of the component-wise operators seems similar to the soft color operators, we remark that the colors generated are different from those in the original image. This is not visually evident, but can be detected when comparing it pixel-wise with the original image. Both color reference algorithms seem to behave similarly, taking into account that they have opposite reference colors for the sake of showing a wider comparison. Both of them order the colors prioritizing their distance with regard to fixed one, usually determining the order regardless of further tie resolving procedures. This is specially relevant when using other operators such as the top-hat or the morphological gradient. However, all of the techniques remove either the eyes or the eyelashes in Fig. 6, since they are almost perpendicular to the non-isotropic structuring element employed.

Finally, Figs. 7 and 8 contain, respectively, the opening and the closing operators. These operators are meant to remove objects smaller than the structuring element and to remove holes between objects that are smaller than the structuring element. This behavior makes them flatten heterogeneous regions, transforming textures into more uniform areas. In addition to it, the opening and the closing should keep the shapes of big objects.

In Fig. 7, we observe how the component-wise approach provides a visually similar image to the original one. Again, the image-dependent morphologies, in Fig. 7e, f, provide significantly different results, modifying the blue region the former and the red region the latter. We remark that only the soft color operators employ the non-binary structuring element, since it is the only paradigm that can meaningfully handle it. The soft response provided by the non-binary structuring element provides both sharp contours and smooth regions.

With regard to the closing operator, in Fig. 8, most alternatives treat the white background as the object to be closed, except for the color reference and the random projections algorithms. The histogram-based method, as a matter of fact, is not able to deal with this artificial image and highly distorts the boundaries of the objects. Some approaches in Fig. 8c, d, f do not preserve the shape of the inner squares. The rest of them, including the soft color operators, are able to deal with this artificial image.

6 Analysis and Conclusions

In this paper, we formally presented the basic operators of the soft color mathematical morphology: the soft color erosion and the soft color dilation (Definitions 7, 8), and we also combined them to create other morphological operators. Then, we studied their properties from a theoretical point of view in Sect. 4, and compared them extensively to other color morphologies in Sect. 5. We end the paper with a discussion of the comparison and the strengths and weaknesses of the soft color morphology.

6.1 Analysis of the Comparison

Regarding color preservation, the only morphology falling short is the component-wise morphology. It is not able to control whether new colors are created or not producing unexpected results when combining erosion and dilation. The rest of methods, which deal with vectors instead of independent channels, do not show this weakness.

With regard to interpretability, the soft color and the lexicographic order for VSH morphologies are interpretable: bright objects are enlarged with dilation and shrank with erosion. This, however, is not always the best solution, as can be observed in Fig. 8: the closing operator is assumed to fill the holes within the objects. On the other hand, adaptive orderings fall short in interpretability: depending on the image, they may produce completely different results. For instance, the same object captured under two different backgrounds may be dealt with in a totally different way. Besides, extra care should be taken when processing artificial images, such as in Fig. 8, since they may alter unexpectedly the ordering.

Taking into account the visual appearance of results (that is, how natural the processed images look), almost all methods provide acceptable responses, even the most basic ones. With a closer look, as in the texture-flattening operator in Fig. 7, the soft color morphology operators provide smoother outputs due to the possibility of using non-binary structuring elements.

Finally, the runtime of methods may also be important in some applications. All of them are appropriate for off-line image processing, although not for real-time image processing. The fastest methods are the ones based on a non-adaptive total ordering—such as morphologies based on bit mixing, color reference and lexicographic orders and the soft color morphology with binary structuring elements. These can leverage the lattice-based definition of erosion and dilation. The component-wise morphology consists on replicating this same operator once per channel, multiplying thus the computational complexity by a fixed constant. The soft color morphology operators do not replicate the same operator several times, but whenever a non-binary structuring element is used, we must evaluate the fuzzy conjunction (or fuzzy implication function) several times. Computationally, this is roughly equivalent to multiplying the computational complexity of the lattice-based definition by as many different values as the structuring element has. Lastly, the adaptive orderings studied here are the most time-consuming methods. They spend the majority of time creating the adaptive ordering, which has to be done once per image. The histogram-based method proposes a solution to this problem, consisting on reusing the same histogram if the processed image is similar to a previously computed one from a statistical point of view.

6.2 Strengths and Weaknesses of the Soft Color Morphology Operators

The interpretability of the operators is of paramount importance. Mathematical morphology encompasses operators that manipulate the shape of objects, and thus they must implicitly define what is object and what is background in an image. This distinction is harder in color or multivariate images than in grayscale or binary images. Our approach is to consider that pixels that belong to the background have higher values in the first channel. This is very intuitive for color images encoded with the CIEL${}^*a{}^*b{}^*$ color space.

On the other hand, these operators can also process images encoded in other color spaces and general multivariate images—like hyperspectral imagery with a large number of channels. However, in this case the practitioner must choose the channel that best distinguishes the objects of interest, or estimate this information from a series of channels (for instance, with the histogram-based approach by Benavent et al. [5]).

The soft color morphology operators fulfill several interesting theoretical properties. Firstly, they preserve the chromatic information of pixels when using the CIEL${}^*a{}^*b{}^*$ color space for color images. Secondly, they can be tuned to preserve full colors, simply by using a binary structuring element. We remark that creating new colors may be desirable in some situations like morphological filtering, where smoothness is favored over realism, although it may be unacceptable in other application fields. If color preservation is pursued, the soft color erosion and dilation become equivalent to the lattice-based definition of dilation and erosion, with an order that is highly interpretable to deal with color images. Thirdly, these operators generalize the fuzzy mathematical morphology ones, which ensures the monotonicity/antimonotonicity of erosion and dilation, among other properties. This is also responsible for the soft outputs achieved with non-binary structuring elements. However, the soft color closing and opening do not inherit the idempotence from the fuzzy mathematical morphology operators. This, which is due to possible ties when finding a maximum or minimum color (see Definition 6), is not a big problem in practice since ties are infrequent in natural images.

The computational complexity is comparable to that of the fuzzy mathematical morphology for grayscale images, which is low but still greater than that of the lattice-based definition of mathematical morphology. On a CPU Intel^© Core^TM i5 @ 3.10 GHz, the time to process one soft color dilation or soft color erosion ranges from 81 ms ($256\times 256$-pixel images) to 1.56 s ($1024\times 1024$-pixel images) with MATLAB^®code, using the minimum conjunction, the Gödel implication and a $5\times 5$-pixel Gaussian-shaped structuring element. This performance could be greatly improved with parallelization and GPU-optimized code, further achieving performances enough to be embedded into real-time systems, as can be seen in [6] in which an equivalent complex operator achieves such a real-time performance.