Unsupervised segmentation and elm for fabric defect image classification

Abstract

In order to solve the problem of low accuracy and efficiency for surface defects in common woven fabrics, a novel fabric defect classification method is proposed based on unsupervised segmentation and ELM. The classification method is divided into four steps including defect segmentation, feature extraction, ELM classifier training, and Bayesian probability fusion. Firstly, an unsupervised segmentation is presented for the Grayscale fabric defect image after preprocessing. Secondly, geometric and texture features were extracted by using the segmented image and the undivided Grayscale image. Then, features and labels in fabric defect images are considered as training sets to train the ELM classifier. Finally, the input fabric defect image is classified by the trained ELM classifier and the Bayesian probability fusion method. Experimental results show that the proposed method can classify the fabric defect image with high accuracy and efficiency that can better meet the requirements for practical applications.

1 Introduction

Clothing is currently the fastest growing shopping category on the web. Compared with the traditional mode, online garment marketing has greater market value and economic benefits. With the developments in customer demand on fabric variety in fashion market, fabric defect detection is central to automated visual inspection and quality control in textile manufacturing. It has been widely appreciated that improving the effectiveness of fabric defect detection is a common demand in the apparel industry. However, the high return rate and the price of loss due to the defects in fabric prevent the benefits of companies and further developments [33]. Currently, there exist more than 70 established categories for fabric defects defined by the textile industry [46]. Due to the immaturity of production techniques or yarn quality, many defects are generated in the fabric production process, such as broken warp, broking weft, wrong weft and warp, etc. And the sewing machine during the process of garment manufacturing results in the defects which involves crease, scratches, shearing, crease, and dirt. Most of these defects are caused by machine malfunctions, yarn problem, and stain of oil caused by the knitting device. Therefore, it is necessary to develop automatic inspection which can enhance the accuracy and efficiency of fabric defect detection for better productivity and improving quality of fabric as well.

Fabric defect classification plays an important role in the quality control of textiles and has historically been achieved via visual inspections by skilled workers. A great deal of work has focused on improving the accuracy and efficiency of fabric defect classification. Traditionally, fabric inspection is relied on human manipulation and experience [37], which suffers from high labor cost and low efficiency. Although the machine detection and efficient system [8] can be used to classify fabric defect. With the fast development of textile industry, fabric texture becomes more and more diverse and complex. As a result, texture has attracted wide attention from various research including 3D surface reconstruction [24] and 2D texture [31]. Jian et al.[26, 29] have proposed efficient methods for illumination-insensitive texture discrimination, which broadly improves the performance for texture classification. However, due to the improvement of weaving techniques, defects are small on such fabric surface than before, which makes the fabric inspection task difficult. Basically, fabric defects can be categorized into 3 groups including defects in the warp direction, defects in the weft direction, and defects with no directional characteristics. Therefore, it is difficult to develop new automated fabric inspection models with higher detection accuracy and efficiency.

Numerous fabric defect detection methods by computer vision and pattern recognition techniques have been proposed in the last three decades [33, 46]. Gray relational analysis [36] was able to recognize fabric defects. Wavelet transform coefficients [18], Fourier transformation [19], Gabor filters [23], and redundant contourlet transform [50] addressed fabric inspection in the spectral domain. Since these methods recognize defects through feature extraction from normal or abnormal fabric texture, the sensitivity of detection can be influenced when the defects are very small and of low contrast. Recently, Ng et al. [45] have used energy minimization to yield good results for defect localization and be applied practically to all fabric types. However, it is not efficient for detecting defects with small contrast in the image (e.g., oil stains).

To our knowledge, the major challenging issues for fabric defect detection include the complexity of fabric textures, the large diversity of defect types, and the accuracy of defect classification. To tackle these issues, we introduce a method for fabric defect image classification through unsupervised segmentation and Extreme Learning Machine (ELM) to balance the efficiency and accuracy. The pipeline of our method is sketched in Fig. 1. The proposed approach includes four main parts: defect segmentation, feature extraction, ELM classifier training, and Bayesian probability fusion. Inspired by the fabric defect detection method based on the dictionary learning framework [58], we present defect segmentation to learn a dictionary from a test image itself. First, an unsupervised segmentation is used to realize the defect segmentation of the fabric. Contrarily to using adaptive template, we focus on unsupervised segmentation which is useful to facilitate the geometric feature extraction. Then we apply ELM as the fabric defect classification. ELM is a new algorithm based on a Single Hidden Layer Feedforward Networks (SLFNs) [21] which has fewer optimization constraints due to its special separability property for classification, and tends to achieve better generalization performance at very high learning speed as compared to traditional SVM. Finally, combined with Bayesian probability fusion method, the fabric defect classification is achieved by training the one-against all (OAA) ELM classifier. The main contributions of this work are as follows.

• Address an unsupervised segmentation method of fabric defect image to avoid human intervention and requiring a large number of non-defective fabric images.

• Propose an ELM classification based on geometric and texture features combined with Bayesian probability fusion method to classify fabric defect images.

• Use multi-resolution wavelet packet decomposition and Hu invariant moments instead of a single feature to improve the classification accuracy.

• Design a sort of objective and subjective measurements for the quality evaluation of our framework, and demonstrate that it outperforms the state-of-the-art methods.

Fig. 1

The pipeline of our four-stages approach, including defect segmentation, feature extraction, ELM classifier training, and Bayesian probability fusion. The key idea of this work is to present an unsupervised segmentation and ELM method for fabric defect classification. We will demonstrate its effectiveness in the following sections

The remainder of this paper is organized as follows. Section 2 presents a brief literature review on fabric defect detection methods. In Section 3, the framework of the proposed fabric defect segmentation is introduced. Section 4 presents our method for Fabric defect classification. Then in Section 5, experimental results and related analysis are provided. Finally, conclusion is given in Section 6.

2 Previous work

Automatic defect detection has gained increasing attention in engineering research. In this section, we firstly review related works in fabric defect detection and recognition. Then, we will review recent progress in defect feature extraction and classification in detail.

2.1 Fabric defect detection and recognition

Referring to the related surveys [33, 46], the most common methods for fabric defect detection can be classified into four groups including statistical, structural, model-based and spectral approaches. Statistical methods tend to distinguish defects through feature analysis of standard textile texture using regularity and local orientation [7]. However, it is very difficult to discriminate small and blurry defects which do not change the average grey-level value of an image. Structural methods claim that texture can be decomposed into a set of textural primitives [39]. Model-based approaches stress that the relationship between pixels in a textural image can be modeled by a predictive mode [18]. Gaussian Mixture Model was applied to acquire the dependencies between wavelet coefficients within each sub-band of wavelet decomposition [32]. In spectral methods [6], fabric defects were extracted by detecting abnormal value in the three dimensional frequency spectrum. The method [1] used wavelet transform combined with threshold segmentation to detect and classify fabric defects involving rough and wrinkled region, which has a few limitations. Hu et al. [18] proposed a Hidden Markov Random Field Model in wavelet domain to detect texture defects combined with threshold segmentation. Recently, Jing et al. [30] proposed a supervised fabric defect detection method with Gabor filters. However, the selection of appropriate parameters of both wavelet transform coefficients and Gabor filters becomes the most challenging task in defect detection issue.

Regarding Wavelet decomposition, fabric defect detection can also be modeled as saliency detection. Many state-of-the-art methods for face recognition were introduced in [49]. And in [25], an efficient hierarchical scheme with a face detector and a wavelet-based saliency map was used for accurate facial-feature detection and localization. Later, a visual-attention-aware model [27] was proposed for salient-object detection based on a bottom-up mechanism. However, these methods cannot be used directly to solve fabric defect recognition. Cao et al. [5] introduced an unsupervised segmentation method that uses least-squares regression based on prior knowledge to detect defects in various fabric textures. The low-rank representation (LRR) technique [38] was used for robust fabric defect detection. Instead of transforming an image into the spectral domain, sparse dictionary reconstruction for textile defect detection was introduced in [56]. However, the sparse representation was based on the basic regression model, which is not suitable for eliminating scattered noise (e.g. spot defects). Consequently, an over-complete dictionary [57] was trained from the testing image itself, but the algorithm was implemented on the original image and its rotated images with high computation cost.

Then, Zhou et al. [58] focused on a fabric defect detection via dictionary learning framework, which is quite similar to template matching but using an adaptive template, such a dictionary is able to approximate training samples well through a linear summation of its elements. But it requires huge amount of training data to ensure the accuracy of the classifier. Moreover, subtle fabric defects are usually too small to be discriminated just by the first and second order statistical features of the image. To overcome the computation problem, a new method [60] based on local patch approximation was presented to address automated defect segmentation on textile fabrics. In contrast, the proposed method adopts unsupervised scheme without the need of reference images or prior information during the whole detection process, which image patch is approximated by dictionary learned from a testing sample in the least squares sense. Different from the method proposed by Zhou et al. [58], we use a local patch approximation method to realize unsupervised fabric defect segmentation, which is devised to identify defects in a fully unsupervised manner without any prior information.

2.2 Feature extraction

A great deal of studies rely on feature extractions of normal fabric texture or fabric defects. The most challenging work in these methods is to find the appropriate features that can be adapted to different types of fabric texture and defects, which possibly influences the accuracy of the detection model [45]. Fabric defect detection schemes can be loosely categorized into feature extraction and non-feature extraction approach. Typical feature extraction methods include grey-level co-occurrence matrix [36], Fourier analysis [6], spectral domain methods [48], and wavelet transform coefficients [61]. Recently, Jian et al. [28] proposed a novel hierarchical-local-feature extraction scheme for content-based image retrieval which avoids the complex image segmentation. But this feature extraction method is not suitable for fabric image with local defect. Although feature extraction is of great importance in fabric detection and these feature extraction approaches mainly rely on feature selection, it can’t guarantee the optimality of features used in case of the absence of defects in training process.

On the other hand, Gabor filter is effective for detecting and classifying fabric defects as the non-feature extraction detection scheme [52]. Bissi et al. [4] proposed the Gabor filter and PCA method that depends on complex and symmetric Gabor filter group, realizing the automatic detection of uniform and structured fabric defect. However, it can not meet the demand in multi-scale and multi-directional filtering. The approach [2] proposed by Anitha et al. can achieve feature extraction respectively, which combined with independent component analysis and vector quantization principal component analysis by using Gabor wavelet network to analyze input fabric defect images. However, this method is only effective for defects with patterned fabric. To improve these schemes, a wavelet packet decomposition based on multi-resolution method [35] was proposed. The key idea of the method is to decompose the low-frequency and high-frequency information, which is beneficial to the extraction of texture features of the fabric defects. However, the detection performance heavily relies on matching of filter parameters and the adjustment to the property of a specific defect, e.g. the scale and orientation of a defect. Besides, a prior knowledge (e.g. templates or defects) is still required for optimizing filters. Different from the existing defect segmentation methods [58] with reference samples needed, our motivation is inspired by the method [60] with a fully unsupervised manner, the dictionary learned in the least squares sense can admit a decent approximation to the local fabric texture in patch-level. In addition, the proposed method can avoid the problem of feature selection compared to the classic feature extraction-based methods, identifying defects in sense of least squared error.

2.3 Defect classification

Most existing methods of image classification use traditional Support Vector Machines (SVM) as the image classifier [12]. Typical classifier methods for fabric detection are mainly based on neural network and SVM classifier [17]. Nasira et al. [44] used gray-level co-occurrence matrix to extract fabric defect texture features, and then combined with the artificial neural network to classify four kinds of fabric defect types: missing end, broken weft, hole and oil stain, which solved the problem of low classification accuracy, but the feature extraction was expensive. Based on the optimal wavelet packet decomposition tree [35] and neural network classifier, knitted fabrics were classified into holes, oil stain, scratches, coarse end, tight warp, missing needles, which can effectively improve the efficiency of texture feature extraction, but ignore the fabric defect geometric features. The propose methods [10, 13, 39] based on SVM classifier has used to classify fabric defect. Although the fabric defect classification based on SVM classifier enhances practical performance, the accuracy of defect classification still needs to be improved further. Since the features extracted by these methods are single and the classifiers are computationally expensive. Meanwhile, the SVM classifier usually suffers from the high computational cost and the large number of parameters to be optimized, which lead to the low accuracy and efficiency of classification.

To solve these issues, Zhang et al. [53] proposed fabric defect classification using radial basis function network. Mottalib et al. [43] applied statistical methods to extract the shape features and used simple Bayesian classifier to classify fabric defect images, which can provide better classification accuracy. To our knowledge, a new learning algorithm called extreme learning machine (ELM) for single-hidden layer feedforward neural networks (SLFNs) in [21] randomly chooses hidden nodes and analytically determines the output weights of SLFNs. Numerous attempts applied ELM classifiers to improve the accuracy of image classification. Lu et al. [41] proposed a multi-category classification method based on extreme learning machine (ELM) for generic images. It not only effectively solved the multi-categories classification problem but also reduced the optimization of the classifier parameters. Zhao et al. [55] used the probability extreme learning machine approach of one-to-one strategy to solve the multi-category classification problem, which can improve the classification efficiency and accuracy. And a signature based on ELM for texture classification was introduced in [31], which outperforms high discriminative performance. When using ELM classifiers, the multi-class problem is decomposed into two-class problem using the One-Against-All (OAA) and One-Against-One (OAO) schemes [47] respectively. Specifically, we also explore the One-Against-All (OAA) method which can decompose a multi-class problem into a set of two-class problem. Then, a Bayesian probability based fusion method is proposed to combine the prediction results of each ELM classifier.

In summary, in view of the accuracy and efficiency, we present a framework of unsupervised segmentation and ELM for fabric defect image classification. Instead of over-complete dictionaries, an unsupervised algorithm using local patch approximation for fabric defect segmentation is more effective in providing relevant structure information for reducing the overall computation cost and enhancing the performance. By means of ELM classifiers combined with Bayesian probability fusion, it can achieve performance at high speed, thereby improving the accuracy of the classification fabric defects.

3 Fabric defect segmentation

Fabric defect classification benefits from automatic segmentation. The supervised approach [34] utilized data-driven techniques to segment a given image with a substantial number of defect-free fabric images. But when the scale of training images decrease, the performance of segmentation results drops dramatically. Without enough defect-free fabric images, the unsupervised segmentation approaches [19] based on fourier analysis and wavelet shrinkage were proposed. To evaluate the effectiveness of these presented approaches, semi-supervised learning method addresses the case where the labeled data is sparse. Mak et al. [42] proposed a semi-supervised filter selection method based on the analysis of spectral feature of textures for fabric defect segmentation, which can automatically find a well matching real Gabor functions in every level of the pyramidal decomposition. Semi-supervised method overcomes the difficulties of requiring a large amount of labeled meshes and the inability to use unlabeled meshes. Essentially, there is another way to alleviate the difficulties of requiring a large amount of labeled data, which is the weakly supervised learning method [16, 54]. Instead of using traditional supervised or semi-supervised learning methodology, these approaches can substantially reduce the human labor of annotating training data while achieving the outstanding performance.

Based on our observation, the locations and sizes of the contained defects vary randomly when a piece of fabric leaves the production line in the textile industry. Different from the existing defect segmentation methods with reference samples needed, we focus on unsupervised segmentation method for segmenting local defects on textile fabrics based on local patch approximation. An example of fabric defect segmentation is illustrated in Fig. 2. The process of fabric defect segmentation involves three steps including patch extraction, dictionary learning, and defect segmentation. Our method firstly attempts to use patch extraction to represent fabric defect images in patch-level. Then dictionary learning is presented to eliminate abnormal patches. Finally, an abnormal map is constructed based on the approximate difference between the defect patch and the normal patch, which can segment the defect area from input fabric image.

Fig. 2

An example of fabric defect segmentation with three steps including patch extraction, dictionary learning, and defect segmentation

3.1 Patch extraction

The unsupervised segmentation method is as follows. Firstly, the preprocessed grey-scale image I of a given image is represented as an image patch W × H in the sense of least squares. Each image patch is represented as a column vector connecting patch columns, and an overlap partition is used to extract the patch. As shown in Fig. 2, let the patch be a size of p × p and patches are extracted from the overlapping division. To eliminate the anomalous elements in the learned dictionary, it is necessary to reduce the number of anomalous patches. Given the image patches {x_i} which collected from the test samples, data matrix can be represent $\mathbf {X}=[{{\mathbf {x}}_{1}},{{\mathbf {x}}_{2}},\cdots ,{{\mathbf {x}}_{i}},\cdots ,{{\mathbf {x}}_{n}}],{{\mathbf {x}}_{i}}\in {{\mathbb {R}}^{p}}$, where p represents the dimension of patch x_i for image I, n is the total number of the patch x_i for image I (that is, containing n column vector of dimension is p). The Euclidean distance E(i) of image patch x_i to data center x in data matrix X can be expressed as follows:

$$ { E(i)=||{{\mathbf{x}}_{i}}-\mathbf{x}|{{|}^{2}}}, $$

(1)

where x is the value of average {x_i}. If the distance E(i) is larger than the threshold T_h, the image patch x_i in data matrix X is eliminated. Taking the smaller proportion of defective area into consideration, we eliminate the total number of patch 0.25 [40], as far as possible to eliminate abnormal patches.

3.2 Dictionary learning

After eliminating the outliers of data matrix X of an image patch, learning a linear combination from image patches is key in sense of least squared error. Given a data matrix X, we need to seek such a linear basis D which can be defined as

$$ {(\mathbf{D},\mathbf{a})=\arg \min \sum\limits_{i = 1}^{n}{||{{\mathbf{x}}_{i}}-\mathbf{D}{{\mathbf{a}}_{\text{i}}}||}_{2}^{2}}. $$

(2)

Where a_i is the vector of coefficients for each x_i of the dimensions k. The above equation produces a dictionary, each point in X of which is represented by the smallest error. Learned from X_c, it contains a dictionary of elements k that expressed as ${{\mathbf {D}}_{\text {t}}}=[{{\mathbf {d}}_{1}},{{\mathbf {d}}_{2}},\cdots ,{{\mathbf {d}}_{i}},\cdots ,{{\mathbf {d}}_{k}}],{{\mathbf {d}}_{i}}\in {{\mathbb {R}}^{p}}(i\le k<n)$. Since the outliers have been eliminated, the learned dictionary D_t captures only the texture features of the area of the non-defective fabric. Aiming at the repetitive spatial structure of fabric texture [38], the linear basis was learned from the image patches after eliminating the discrete values. And the dictionary elements was obtained, which can constitute the dictionary D_t representing the patch level normal fabric texture in the lower dimension subspace to improve the efficiency. Thus, X_t is the outlier removed data matrix and D_t is the dictionary learned from X_t with k dictionary elements.

3.3 Defect segmentation

Because the learned dictionary D_t only can capture the normal texture structures of the fabric defect sample, it will lead to the approximate residual error of the patch level. To our knowledge, the approximation error derived from the approximation with D_t for normal and abnormal patches is able to use for defect discrimination in [38]. Then constructing an abnormal map through the original image patch x_i and the pixel difference S(i) based on approximate patch $\overset {\wedge } {\mathop {{{\mathbf {x}}_{i\psi } }}}$is presented to solve substantial approximation error on defective regions. Considering the neighborhood pixels, the defect segmentation operation will be performed to segment the defective regions from the abnormal map, and the pixel difference S(i) can be calculated as follows,

$$ { S(i)= 1-\exp (-\gamma ||{{\varpi} ^{\text{T}}}({{\mathbf{x}}_{i}}-\overset{\wedge} {\mathop{{{\mathbf{x}}_{i\psi} }}} )|{{|}^{2}})}. $$

(3)

Where γ by user-defined is used to control the sensitivity of segmentation parameters, the weight ϖ^T is used to reduce the impact of the pixels from the center pixel. The patch x_i away from the center pixel ψ is calculated by two-dimensional Gauss function which is defined as $f(x,y)=\exp (-\frac {({{x}^{2}}-{{y}^{2}})}{2{{\sigma } ^{\prime }}^{2}})$, where standard deviation σ′ determines the curve, the weight ϖ^T is similar to a low-pass filter that has a maximum center weight. With the increase of the radius, weight will reduce.

Finally, the threshold operation is used to segment the defect region from the abnormal map. Binary segmentation results B_W are calculated as [57]

$$ {{{B}_{W}}(i,j)=\left\{ \begin{array}{ll} 0,{{\mu} _{1}}-c\cdot {{\sigma} _{1}}<S<{{\mu} _{1}}+c\cdot {{\sigma} _{1}} \\ 255, otherwise \end{array} \right.}, $$

(4)

where μ₁ and σ₁ are respectively the pixel mean and standard deviation in the residual image, and c is a preset constant. i and j are locations of pixels, and E is the residual image. The abnormal map can differ defective regions from background than simple subtraction map, and highlight the anomalies in normal regions, which will significantly facilitate defect segmentation.

The pseudo code of fabric defect segmentation method is given in Algorithm 1.

4 Fabric defect classification

After segmentation, we extract the geometric features of the segmented fabric defect images and the texture features of the grey-scale images respectively. Finally, combined with Bayesian probability fusion, the fabric defects are classified by ELM-OAA classifier.

4.1 Feature extraction

Choosing the appropriate features is of importance for fabric defect classification. On the one hand, we use the Hu invariant moment [59] to extract the geometric features of the fabric defects G″ after segmentation. On the other hand, the fabric defect image G′ is decomposed by wavelet packet based on optimal wavelet packet technology. Then the texture features of the fabric defect image can be obtained by calculating Shannon entropy.

Given the segmented fabric defect image classification, the geometric moment m_pq of p + q order can be formulated as

$$ {{{m}_{pq}}=\sum\limits_{x^{\prime}}{\sum\limits_{y^{\prime}}{{{x}^{'p}}{{y}^{'q}}F({{x}^{\prime}},{{y}^{\prime}}),\quad p,q = 0,1,2,...,n}}} , $$

(5)

where F(x′, y′) is the image gray value function, x′ and y′ represent the image coordinates. Due to the invariance of the geometric moment m_pq, the center moment can be obtained by the following formula:

$$ {{{u}_{pq}}={{\sum\limits_{x}{\sum\limits_{y}{(x-\bar{{x}^{\prime}})}}}^{p}}{{(y-\bar{{x}^{\prime}})}^{q}}f(x,y),p,q = 0,1,2,\cdots ,n}, $$

(6)

where $\bar {{x}^{\prime }}={{{m}_{10}}}/{{{m}_{00}}}$ and $\bar {{y}^{\prime }}={{{m}_{10}}}/{{{m}_{00}}}$ respectively. Then, the normalized central moment η_pq is defined as

$$ {{{\eta} _{pq}}=\frac{{{u}_{pq}}}{{{u}_{00}}^{r}},p+q = 2,3,\cdots}. $$

(7)

The central moment of $r=\frac {(p+q)}{2}$ can describe the geometric area instead of describing the invariance of rotation. In practical applications, the fabric defect not only appears in the fabric anywhere, but also has the invariance of rotation. Therefore, Hu’s seven moment invariants ϕ_i(p + q ≤ 3) can satisfy the invariability of the translation and the rotation.

$$ {\left\{ \begin{array}{ll} & {{\phi} _{1}}={{\eta} _{20}}+{{\eta} _{02}} \\ & {{\phi} _{2}}={{({{\eta} _{20}}-{{\eta} _{02}})}^{2}}+ 4{{\eta} _{11}}^{2} \\ & {{\phi} _{3}}={{({{\eta} _{30}}-3{{\eta} _{12}})}^{2}}+{{(3{{\eta} _{21}}-{{\eta} _{03}})}^{2}} \\ & {{\phi} _{4}}={{({{\eta} _{30}}+{{\eta} _{12}})}^{2}}+{{({{\eta} _{21}}+{{\eta} _{03}})}^{2}} \\ & {{\phi} _{5}}=({{\eta} _{30}}-3{{\eta} _{12}})({{\eta} _{30}}+{{\eta} _{12}})[{{({{\eta} _{30}}+{{\eta} _{12}})}^{2}}-3{{({{\eta} _{21}}+{{\eta} _{03}})}^{2}}]+ \\ & (3{{\eta} _{21}}-{{\eta} _{03}})({{\eta} _{21}}-{{\eta} _{03}})[3{{({{\eta} _{30}}+{{\eta} _{12}})}^{2}}-{{({{\eta} _{21}}+{{\eta} _{03}})}^{2}}] \\ & {{\phi} _{6}}=({{\eta} _{20}}-{{\eta} _{02}})[{{({{\eta} _{30}}+{{\eta} _{12}})}^{2}}-3{{({{\eta} _{21}}+{{\eta} _{03}})}^{2}}]+ \\ & 4{{\eta} _{11}}({{\eta} _{30}}+{{\eta} _{12}})({{\eta} _{21}}+{{\eta} _{03}}) \\ & {{\phi} _{7}}=(3{{\eta} _{21}}-{{\eta} _{03}})({{\eta} _{30}}+{{\eta} _{12}})[{{({{\eta} _{30}}+{{\eta} _{12}})}^{2}}-3{{({{\eta} _{21}}+{{\eta} _{03}})}^{2}}]- \\ & ({{\eta} _{30}}-3{{\eta} _{12}})({{\eta} _{21}}+{{\eta} _{03}})[3{{({{\eta} _{30}}+{{\eta} _{12}})}^{2}}-{{({{\eta} _{21}}+{{\eta} _{03}})}^{2}}] \end{array} \right.}. $$

(8)

Due to the large range of seven moment invariants ϕ_k(k = 1, 2, ⋯ , 7), we need com- press the data. The actual moment invariant is calculated by using the following formula:

$$ {{{I}_{k}}={{\log} _{10}}|{{\phi} _{k}}|(k = 1,2,\cdots ,7)}. $$

(9)

The texture features of gray-scale fabric defect images are extracted based on multi-scale wavelet packet transform [11], in which the wavelet packet decomposition images are based on the optimal wavelet packet basis tree. Each image is decomposed into four sub-images $\frac {N}{{{2}^{j}}}\times \frac {N}{{{2}^{j}}},(j = 1,2,\cdots ,J)$. At resolution level j, sub-image pixels are $\frac {N}{{{2}^{j}}}\times \frac {N}{{{2}^{j}}}$. These sub-images not only provide a variety of expressions of different frequency bands of gray-scale images, but also save time for feature extraction. The original gray-level image and sub-images can be regarded as the parent and child nodes of a tree. Shannon entropy of an image N × N is calculated as:

$$ {\varepsilon =-\sum\limits_{m = 1}^{N}{\sum\limits_{n = 1}^{N}{[\omega {{(m,n)}^{2}}]\log [\omega {{(m,n)}^{2}}]}}}. $$

(10)

Where ω(m, n) represents the wavelet packet coefficients of the image. If the entropy of a child node is less than a parent node, each child node continues to be decomposed; otherwise, it will be stopped.

4.2 ELM classifier training

Differs from the conventional neural network theories, a new method in [20] was proposed to show that single-hidden-layer feedforward networks (SLFNs) with randomly generated additive or radial basis function (RBF) hidden nodes can work as universal approximators. ELM is a new algorithm based on Single Hidden Layer Feedforward Networks (SLFNs). Although there are a lot of different versions of ELM in the literature such as ELM-SLFN (Single Layer Feed-Forward Network), ELM-RBF (Radial Basis Function), and Kernel-ELM, and etc. In this paper, we propose using ELM to improve the accuracy of defect classification, specifically for multi-category classification, we also explore the One-Against-All (OAA) method [47] named as ELM-OAA for brevity which can decompose a multi-class problem into a set of two-class problem.

To solve the multi-class classification problem, the ELM classifier [22] uses networks whose multiple output nodes are equal to the pattern categories, and the visual feature s_i of each participating training is the feature f_ki(k = 1, 2) of the extracted fabric defect image. The target output t_i is a m dimensional vector (t₁, t₂, ... , t_m)^T. The training set Z is composed of N visual features s_i involved in the training and the target output t_i.

The training algorithm is as follows. Given a training set $Z=\{({{s}_{i}},{{t}_{i}})|{{s}_{i}}\in {{\mathbb {R}}^{n}},{{t}_{i}}\in {{\mathbb {R}}^{m}},i = 1,...,N\}$, activation function G(x), the number of hidden nodes L, ELM classifier first randomly assigns hidden nodes (a_i, b_i), i = 1, ⋯ , L, and then calculates the hidden node output matrix H, and finally computes the output weight $\hat {\beta } =H+T$.

In order to train the ELM-OAA classifier, we standardize the extracted features of fabric defects for geometric eigenvector f_ki = (f_ki1, f_ki2, ... , f_kin) with n dimensioned textures. And the sample mean vector μ and standard deviation vector σ are calculated as follows

$$ {\mu =\frac{1}{N}\sum\limits_{i = 1}^{N}{{{f}_{ki}},\sigma =\sqrt{\frac{1}{N}\sum\limits_{i = 1}^{N}{{{({{f}_{ki}}-\mu )}^{2}}}}}}, $$

(11)

where N represents the number of eigenvectors and the normalized eigenvectors are expressed as $f_{ki}^{\prime }=\frac {({{f}_{ki}}-\mu )}{\sigma }$.

Given the training set which involves seven common types of fabric surface defect including hole, oil stain, weft stripe, crumple, scratches, pressed mark, and crease, we use 7 binary ELM classifiers (see Fig. 3). Each binary ELM classifier has the same input data s_i and different target data t_i, which is trained independently according to the ELM classifier training algorithm. For x-th binary ELM classifiers, there is an output node whose output value is y^x. The seven binary ELM classifiers generate seven predicted values. Using a single hidden layer feedforward network to connect x-th binary ELM classifier which shares L hidden nodes output neuron weights β^x learned without affecting other weights, We can define the output weight vector β^x as ${{[{\beta _{1}^{x}},\cdots {\beta _{L}^{x}}]}^{\text {T}}}$, where, β^x = (H^x)⁺t^x and t^x is defined as ${{t}^{x}}=[{t_{1}^{x}},\cdots ,t_{{{N}^{x}}}^{x}]$, and N^x is the number of training sets involved in x-th binary ELM classifier. H^x can be obtained from

$$ {{{H}^{x}}=\left[ \begin{array}{ccc} & G({{a}_{1}},{s_{1}^{x}},{{b}_{1}}),\ldots ,G({{a}_{L}},{s_{1}^{x}},{{b}_{L}}) \\ & \quad \quad \ \vdots \quad \quad \ \cdots \quad \quad \quad \ \vdots \\ & G({{a}_{1}},s_{{{N}^{x}}}^{x},{{b}_{1}}),\ldots ,G({{a}_{L}},s_{{{N}^{x}}}^{x},{{b}_{L}}) \end{array} \right]}. $$

(12)

Fig. 3

The illustration for our ELM-OAA classifier training

Since the multiple outputs from the SLFNs corresponding to the each binary ELM classifier represent collective outputs for the 7 classes, thus, we need to integrate them into the final class label for the training image sample. We minimize the total loss by ensuring that the training samples (s_i, t_i) is most consistent with the label category of output y_i, which means the total loss for sample y_i is the minimum for all class labels. The total loss for training samples (s_i, t_i) is calculated as $D(M,y(s))=\sum \limits _{x = 1}^{7}{{\Theta } (M(i,x){{y}^{x}}(s))}$. Where M is the matrix, and diagonal elements are + 1, and other elements are -1. The output value of x-th binary ELM classifier is y^x(s), and θ is the exponential loss function. The final output form of the training samples (s_i, t_i) is as follows

$$ {\tilde{y}(s)=\arg \underset{i = 1,\cdots ,7}{\mathop{\min} } D(M,{{y}_{i}}(s))}. $$

(13)

Finally, the output weights of 7 binary ELM classifier are trained.

4.3 Bayesian probability fusion

To improve the accuracy of classification, we propose using Bayesian probability to fuse each prediction results of ELM-OAA classifier training features, which avoids the way that a single feature attribute directly determines the fabric defect image category. For a fabric defect image to be classified, a single feature ${{\{f_{G^{\prime }}^{k}\}}^{K}}$ is first selected from the geometric and texture features from the fabric defect image. Then we input them into each trained ELM-OAA classifier, which provides a range of [0,1] prediction results as posterior probability $P({{t}_{i}}|{{\{{{\tilde {y}}_{{{G}^{\prime }}}}\}}^{K}})$. In this paper, we assume that different feature conditions are independent, and the posterior probability corresponding to K different eigenvalues is defined as follows:

$$ { P({{\{{{\tilde{y}}_{{{G}^{\prime}}}}\}}^{K}}|{{t}_{i}})=\prod\limits_{k = 11}^{K}{P({{\{{{{\tilde{y}}}_{{{G}^{\prime}}}}\}}^{k}}|{{t}_{i}})}}. $$

(14)

According to Bayesian theory $P({A}/{B} )=\frac {P(A)}{P(B)}P({B}/{A} )$, the probability fusion of multi-features in this paper can be calculated by the formula

$$ { P({{t}_{i}}|{{\{{{\tilde{y}}_{{{G}^{\prime}}}}\}}^{K}})=\frac{P({{t}_{i}})}{P({{\{{{{\tilde{y}}}_{{{G}^{\prime}}}}\}}^{K}})}P({{\{{{\tilde{y}}_{{{G}^{\prime}}}}\}}^{K}}|{{t}_{i}})} . $$

(15)

Where $P({{\{{{\tilde {y}}_{{{G}^{\prime }}}}\}}^{K}})=\prod \limits _{k = 1}^{K}{P{{({{{\tilde {y}}}_{{{G}^{\prime }}}})}^{k}}}$, the prior probability P(t_i) is assumed to be uniform distribution, the final fusion results can be calculated by this form:

$$ { P({{t}_{i}}|{{\{{{\widetilde{y}}_{G^{\prime}}}\}}^{K}})=\frac{P({{t}_{i}})}{P({{\{{{\widetilde{y}}_{G^{\prime}}}\}}^{K}})}\prod\limits_{k = 1}^{K}{P({{\{{{\widetilde{y}}_{G^{\prime}}}\}}^{k}}|{{t}_{i}})}} . $$

(16)

In the same way, for the two ELM classifiers, the final fusion result is calculated as

$$ { P({{t}_{i}}|{{\{\widetilde{L}\}}^{K}})=\frac{P({{t}_{i}})}{P({{\{\widetilde{L}\}}^{K}})}\prod\limits_{k = 1}^{K}{P({{\{\widetilde{L}\}}^{k}}|{{t}_{i}})}}. $$

(17)

The final result is an interpretable posterior probability of the image relevance with respect to the expected value of the ELM-OAA classifier output. The pseudo code of the fabric defects image classification strategy is presented in Algorithm 2.

5 Experimental results and analysis

In order to assess the performance of the proposed method, we firstly give the dataset and implementation details in the experiment, and compare our framework with state-of-the-art approaches. Secondly, we evaluate the unsupervised segmentation and analyze the qualitative and quantitative results. Thirdly, several experiments are performed to analyze feature extraction results. Then, we compare the accuracy of our classification results with other results. To further demonstrate the effectiveness, the runtime performance is presented to aid the analysis.

Dataset

The performance of the proposed fabric defect classification is validated through two dataset: the first one is the popular TILDA dataset [9]. As remarked in the dataset, an image of 768 × 512 in TILDA only covers an area of 37.5 square centimeter around. The second one is our dataset acquired from an apparel factory in Mainland China and scanned from the fabric defect handbook. The dataset obtained from grey-scale processing in the experiment include a training set and a test set, which contain the most common 7 types of fabric defects, such as hole, oil stain, weft stripe, crumple, scratches, pressed mark, and crease that always appear in the textile industry. Table 1 shows the number of input training images, the types of fabric defects, the number of each type of training and test images, and the total number of training and test images. To ensure the accuracy of the classification results, all the images are obtained from the standard textile defects in the textile industry, of which the types of defects are often found in the textile and garment manufacturing industries. In our dataset, images of the real fabric samples were captured at 800 × 800 by a digital camera. The image size is set to pixels, pre-processing for the 8-bit grey scale.

Table 1 Our dataset of the fabric defects images used in the experimental evaluation

Table 2 analyzes the overall testing results by using our proposed approach on TILDA and our dataset, respectively. The four measurements provide the evaluation from different aspects. Precision indicates the percentage of correct alarm during detection. Sensitivity shows the percentage of defective samples that are correctly detected. Specificity manifests the percentage of non-defective samples that are correctly classified as normal. Accuracy is the percentage of correct classification of all testing samples.

Table 2 Performance evaluation of our proposed method on TILDA and our dataset (Unit: %)

Implementation details

All the tests are performed on a computer with an Intel Core i5 3.3GHz CPU and 8GB DDR3 RAM. The major part of the code is implemented on MATLAB software platform.

Comparison with relevant methods

Although the accuracy of fabric defect image classification is inferior to the machine detection and classification, fabric defect image classification can reduce the cost and shortening development process. Our method is inspired by the goal of improving the quality of fabric production and guaranteeing the completeness of clothing fabric. Distinct from several other relevant methods, we present an approach based on unsupervised segmentation and ELM to solve the problem of low accuracy and efficiency of surface defects in common woven fabrics. Table 3 compares the performance of these methods qualitatively.

Table 3 Comparison between our method and the other main relevant approaches

5.1 Segmentation evaluation

We evaluate our unsupervised segmentation method for the analysis of the qualitative and quantitative results. Figure 4 shows the segmentation results of four types of fabric in TILDA. One representative sample was selected as demonstration for each type of defects including holes, oil stain, missing yarn and spot. Most of the fabric defects were well detected, which reveals that the detection model is of high sensitivity and robustness. Given that most of the fabric defects found in a real case might be much smaller than the samples in TILDA, more experiments were conducted on some real samples collected from an apparel company and scanned from the fabric defects handbook to verify the effectiveness of the proposed method for the apparel industry.

Fig. 4

Segmentation results of fabric defects in TILDA dataset. a Hole. b Oil stain. c Missing yarn. d Spot

We also show the segmentation results of 7 types of woven defects of different textures in our dataset. Figure 5 shows a statistical evaluation of the segmentation results for 7 kinds of woven defect types, where the parameter k = 7, patch size is 28 × 28. The input original plain weave fabric defect images are given in Fig. 5a-g (left). The right columns in Fig. 5a-g are the segmentation results by our method.

Fig. 5

Segmentation results of woven defects. a Pressed mark. b Crumple. c Scratches. d Weft stripe. e Oil stain. f Crease. g Hole. The defect area of original fabric image is marked with red rectangle marks

To determine the effect of the segmentation in our unsupervised method, we repeated the experiment by using 21 fabric defect images. Table 4 shows the segmentation results of 7 types of woven defects of different textures. The left column from top to bottom in Table 3 is the original input user fabric defect images marked red rectangle for defect area, and the right column shows the unsupervised segmentation results of our method. Combined with the analysis of the visual results in Fig. 4 and Table 4, our method has better segmentation results on 7 fabric defects under different textures.

Table 4 Experimental segmentation results of 7 types of woven defects of different textures

In this paper, the following three measurements including precision A, recall R, and value F are employed to judge the performance of defect segmentation result. $A=\frac {{{N}_{ri}}}{{{N}_{gti}}}$, $R=\frac {{{N}_{ri}}}{{{N}_{ni}}}$, $F = 2\cdot \frac {{{N}_{ri}}}{{{N}_{gti}}\cup {{N}_{ni}}}$, where N_ri is the positive pixel of segmentation result in the image M_i, and N_gti is truly defective number of pixels in the image M_i, N_ni is the total number of pixels in the segmentation result of image M_i. The precision and recall of our method are calculated at the pixel level. The precision refers to the proportion of the correct defective pixel extracted by our algorithm to the true pixel of the defect. The recall represents the proportion of the correct defective pixel extracted by our algorithm to the total defective pixel extracted by our algorithm. F value is the proportion of pixels extracted from the true defect area to the union which is extracted defective pixel and the pixel of the true defect.

Because two parameters including number of elements in learning dictionary k and patch size p × p are used in our method, we verify the influence of these two parameters by calculating precision, recall and F value of different k and p. The true defect area of all input fabric defect images will be manually marked as a basis for calculating F value and precision.

The average AR curves of all fabric defect samples are shown in Fig. 6. The curve AR shows that the overall performance of our segmentation method is poor and relatively stable when k < 15 and p > 15 (see F value in Fig. 6c). The segmentation performance changes a little when k < 15 and p > 15 (See Fig. 6) and it does not work well in precision when k is too large when k = 15 (see precision in Fig. 6a). The reason for the above phenomenon is as follows. On the one hand, k is too small, and the probability of fitting a potentially defective area is increased when fitting the texture of the fabric, reducing the significance of defects in the isomeric map. On the other hand, p is too small, and the fabric texture can’t be sufficiently captured when the defect size is larger than the patch size, resulting in poor discrimination. According to the analysis of performance parameters, the satisfactory results can be always obtained by using our segmentation method, and it shows certain robustness.

Fig. 6

Average AR curve of fabric defect sample. a Accuracy under the number of different elements K. b Recall under the number of different elements K. c F value under the number of different elements K

To demonstrate the effectiveness of the proposed unsupervised segmentation performance, we compare our method with visual mechanism of wavelet domain [15], supervised segmentation method [30], dimensional empirical mode decomposition approach [38], and dictionary learning framework [58], as shown in Table 5. The four parameters (precision A, recall R, value F and runtime) are used to analysis the performance between our method and others. For the relative methods and ours with the same unlabeled training sets for 7 types of fabric defects, and the fabric defects of D1 to D7 are as follows: D1(Hole), D2(Scratches), D3(Crease), D4(Pressed mark), D5(Crumple), D6(Weft stripe), D7(Oil stain). As shown in Table 5, the average accuracy of our unsupervised method is 92.6%, which is approximately 12.45% better than the supervised approach. And we also tested on some training datasets with label missing or error. Although the recall of ours is low than others, the experimental results demonstrate that our method shows relatively robustness compared with other segmentation approaches.

Table 5 Comparisons between our segmentation method and other several approaches with four performance parameters including precision A, recall R, value F and runtime

5.2 Feature extraction analysis

Geometric and the texture features

To evaluate the accuracy of classification, we have analyzed the extraction for geometric and the texture features respectively. Figure 7 shows a statistical evaluation of the classification results for different types of fabric defect images using geometric features, texture features, and our method. For fabric defects classification task, we repeat the procedure with the defect images including hole, oil stain, weft stripe, crumple, scratches, pressed mark, and crease. Figure 7(left) presents the grey-scale fabric images that to be classified and the classification results of fabric defect image obtained by different feature extraction methods are shown in Fig. 7(right). The red triangle marks illustrate the correct classification results, and the blue ones show the incorrect results. As shown in Fig. 7(right), the classification error will be generated when using single texture or geometric feature with the similar texture in grey-scale images or similar geometric texture after segmentation. However, the satisfactory results can always be obtained by using our method, compared with the extraction for single geometric or texture features.

Fig. 7

The classification results for different types of fabric defect images (left) using geometric features, texture features, and our method

Invariance analysis

Geometric features can distinguish the shapes of fabric defects. However, for many defect shapes with the same contour, such as hole defects and oil stain can only be recognized by texture feature parameters. We also evaluated the accuracy of our method by using three measurements including Roughness(R), Contrast(C) and Orientation(O). As illustrated in Table 6, the R, C and O related to the defect images with different textures are given respectively.

Table 6 Invariance analysis including Roughness(R), Contrast(C) and Orientation(O) for six different fabric defects

Roughness(R)

Table 6 shows there are three sets of the same fabric defects with the same geometric feature values. The values of warp stripe and warp scratch are 12.6012 and 16.1355, respectively. Hole defects and oil stain values are 10.8911 and 11.0143, respectively. The values of missing weft and weft scratch are 15.9428 and 19.1592, respectively. The two values of each group are different, which indicates that roughness measurement can distinguish similar geometric feature values in the same fabric.

Contrast(C)

The contrast value is constantly changing due to the different textures with a variety of colors and patterns on fabric defect images. As seen from Table 6, because the defect area contains both real defects and incomplete repeating pattern units, the pair ratios of warp stripe and warp scratch (0.6306 and 1.0416) are significantly lower than those of hole defects and oil stain (2.9153 and 2.8094). Thus, the contrast value is affected by the patterns of different fabrics. However, the difference between the two contrast values is small when there are two similar defects in the same fabric, such as the contrast values in weft stripe and weft scratch (0.7945 and 0.9999). Therefore, contrast is also an effective parameter to describe texture features of fabric defects.

Orientation(O)

Table 6 also shows the defect images with different texture orientations and their corresponding directional values. The experimental results indicate that there is no clear distinction rule for the overall direction values of the six defect samples. The orientation of missing weft is about 0, indicating that it is almost random. For defects similar to those in the same fabric, orientation cannot be distinguished from defects, such as the orientation in missing warp and warp stripe is 0.0482 and 0.3286, respectively. These features are orientation-invariant for image patch.

Therefore, using our proposed method to extract the features of fabric defect images can satisfy translation and rotation invariance.

5.3 Classification comparisons and results

To demonstrate the accuracy of our classification method, we compared our method with the other classification approaches such as ANN [3], SVM [14], Fuzzy [34], KNN [51], and RBF [53]. We performed the experiments in the same manner with the unsupervised segmentation results. As shown in Fig. 8, the classification accuracy of our method is 91.8%, which is better than the other methods. The accuracy of ANN [3] is 74%, the accuracy of the method based on SVM [14] is 80.1%, the accuracy of Fuzzy [34] is 90.4%, the accuracy of KNN [51] is 88.3%, and the accuracy of RBF [53] is 83.2%.

Fig. 8

Analysis of accuracy between our classification method and other approaches

Figure 9 depicts the statistic and analysis of final classification results for 7 types of fabric defect images with different textures. The fabric textures contain plain, twill, dot, grid and stripe and the defects include hole, oil stain, weft stripe, crumple, scratches, pressed mark, and crease. As shown in Fig. 9, the classification accuracy differs from the same defect of the fabric with different textures and the different defects of the fabrics with the same texture. Combined with fabric defect image content and segmentation results analysis, the reason is twofold. Firstly, the accuracy of segmentation of certain defects in woven fabrics is low with certain textures, which leads to the low accuracy of feature extraction. Secondly, the accuracy of feature extraction is low with certain textures, leading to the errors in the classification of these fabric defects.

Fig. 9

The classification accuracy of 7 types of fabric defects in this paper

In the next test, the experimental runtime is used to illustrate the efficiency of our classification method compared with other classifiers such as SVM-OAO, SVM-OAA, and ELM-OAO. Table 7 lists the average training CPU time of the same test examples. The training time of SVM-OAO classifier is 13.9604(s) and the SVM-OAA is 120.1606(s), the time-consuming process greatly limits the application of these methods. And the training time of ELM-OAO is 5.8601(s). The runtime by using our ELM classification and OAA decomposition method is 3.7309(s) with high efficiency.

Table 7 Performance statistics of the average training CPU time for the test examples. (Unit of time: s)

6 Discussions and conclusions

How to detect and classify the surface defects in common woven fabrics is a challenging problem. The entire process of fabric defect classification is labor and computation intensive. In this paper, we present a novel fabric defect classification method to balance the efficiency and accuracy, which includes defect segmentation, feature extraction, ELM classifier training, and Bayesian probability fusion. The performance of the proposed defect detection model was evaluated on the basis of TILDA database and some more real fabric samples. First, an unsupervised algorithm using local patch approximation for fabric defect segmentation is proposed to reduce the overall computation cost. Then, with a fully unsupervised manner, we use a decent approximation to the local fabric texture in patch-level.Last, we develop a classification method based on the trained ELM classifier and the Bayesian probability fusion to classify the input fabric defect images to efficiently improve the accuracy of the classification fabric defects. A series of results demonstrates its effectiveness on the detection of defects of various shapes, sizes and locations.

Extensive experiments were conducted to validate the robustness of the proposed method compared with other segmentation approaches, such as supervised segmentation method [30], dictionary learning framework [58], visual mechanism of wavelet domain [15] and dimensional empirical mode decomposition approach [38]) in Table 5. The average accuracy of our unsupervised method is 92.6%, which is approximately 12.45% better than the supervised approach. A statistical evaluation of the classification results were tested for different types of fabric defect images using geometric features, texture features, and our method. We also evaluated the accuracy of our method by using three measurements including Roughness(R), Contrast(C) and Orientation(O) to show that the features are orientation-invariant for image patch.

Distinct from several other relevant methods [5], [30], and [53] for fabric defect image classification, our method focuses on improving the quality of fabric production and guaranteeing the completeness of clothing fabric. Furthermore, the classification comparisons with other approaches such as ANN [3], SVM [14], Fuzzy [34], KNN [51], and RBF [53], and the average training CPU time of our ELM classification and OAA decomposition method is 3.7309(s) with high efficiency. And the classification accuracy of our method is 91.8%, which is better than the other methods.

Limitations

Our system has a number of limitations, which point out the direction of future study. Firstly, our method is only able to detect and classify the fundamental common woven fabrics and is not suitable for all types of woven fabric defects. Secondly, we just consider TILDA database and limited dataset of training fabric defect images for segmentation, feature extraction and classification. Additionally, the approach cannot be considered to detect and classify for all types of fabric defects.

Future Works

We hope that the proposed method for fabric defect image classification based on unsupervised segmentation and ELM encourages further research in the future work. We would like to extend our fabric defect classification to more fabrics and address the limitations in the current method as described above. More investigations and measurements are required, in particular for large set of training images to arrive all types of fabric defects fully automatically. Trying to address the types and textures of fabric defects is our key points. We will focus on classification using deep learning and adopting efficient classifiers. Furthermore, applying weakly supervised learning method for segmentation will be our interesting direction in the future. We are also interested in extending our segmentation and classification method to more applications.