State of the Art and a New Methodology Based on Multi-agent Fuzzy System for Concrete Crack Detection and Type Classification

Abstract

Routine inspection and automatic distress detection and classification are critical for civil infrastructures such as bridges. The main subject of this paper is to provide brief review a multi-agent fuzzy system (MAFs) based on image analysis for the detection and classification of, various types of cracking in concrete elements. For this purpose, the combination of fuzzy inference systems has been developed as an autonomous intelligent agent in the center of a multi-agent system (MAS) which communicate, and exchange information with each other. This work is presented in two main sections, (1) the crack recognition system and (2) the type detection system that both of them designed based on MAS fuzzy systems. The first module is a binary classification agent that made of 5 inputs, one output, 11 rules. This agent receives an image and classified it into two groups: crack and non-crack. The second module, which is made of 8 inputs, three output, 20 rules, and used for type classification (individual, pattern, and random). The input of this module is the images that were classified using the first module into the crack group. Particle swarm optimization has been used to find the optimal values of membership functions coefficients. The optimized results of multi-agent modules are compared with other methods. After an experimental characterization and optimization of modules, the MAFs are tested on various concrete distresses. The results show a high potential of MAFs for crack detection and classification. Analysis of the results showed that accuracy of detection, classification can be improved by 4% and 5%, respectively, with MAFs. This method enhances the speed, accuracy, and has higher precision, which indicates the satisfaction and reliability of the MAFs. Also, besides this system has high computational power to detect and classify complex cracking patterns in bridge components.

1 Introduction

Almost nearly all road and transport agencies around the world, are struggling with the problem of quickly and accurately identifying infrastructure failures such as cracking for infrastructures like buildings [1,2,3], roads [4,5,6], tunnels [7,8,9] and bridges [10,11,12]. More than one million bridges per year in the US and EU visually inspected [13].

Recently, published statistics show that More than 365,000 bridges are in service to Iranian National roads. The percentage of bridges with spans up to 6 m, 6–10 m, 10–20 m, 20–30 m, up to 30 m are respectively 94.2, 3.2, 1.3, 0.3 and 0.3%. In the United States, approximately 615,000 bridges are in service, and 235,020 (38%) of U.S. bridges have identified repair needs. American Road & Transportation Builders Association (ARTBA), reported, “4 out of 10 U.S. bridges need to be replaced or repaired, including 1 in 3 bridges on the Interstate”. National law in different countries requires that famous bridges be repeatedly checked for condition evaluation [14]. Field visit or visual inspection is one of the most common methods of bridge assessment, based on this information, the appropriate maintenance method before reaching critical condition is selected [15]. However, visual inspection and site audit, includes high labor costs, dangerous problems, safety, and makes repeatable and unreliable results [6].

Assistance such can be very effective and efficient [16]. The fast, economic, and safe assessment of bridge condition is a critical task for inspection and evaluation [17]. There are different types of distress in concrete structures, can be classified as cracking, scaling, spalling, reinforcement corrosion, pop-outs, and delamination. Among these distresses, cracking is the most significant defect and play a crucial role in M&R and decision making. In addition, automatic detection and analysis is an essential aspect of time, cost, safety, and accuracy. The detection and classification of a crack are important for; (1) determining the cause of distress and extract the second parameters that help to select the most suitable materials and methods [18]. (2) The Structural Health Monitoring (SHM) and Assessment (SHA) indexes represent the overall current condition of the structure. The quantitative index is the parameters that help decision-makers’ strategic managements in-network level for preventive maintenance, prioritization, and rehabilitation and project level for treatment selection (Fig. 1).

Fig. 1

Comprehensive comparison of documents in recent years based on keyword BHM

From the beginning of 2000, the growth rate of articles in this regard (bridge AND health AND monitoring) is a 4-fold increase to the beginning of 2019, according to Scopus analysis (see Fig. 2). The present section comes out from several-steps:

Fig. 2

The steps of review task

(1) Comprehensive comparison of issue; (2) extraction of publications; (3) filtering and collect related publication (make an article bank); and (4) article bank analysis.

Scientists have studied various areas and techniques, and these methods can be classified. The latest studied on bridge evaluation and condition inspection is shown in, and they were considered in 2000–2019. For nearly all the procedures, three steps have to be taken into databases, the Image Acquisition component (IAC), Image Processing component (IPC), and Image Interpretation component (IIC). This section presents a brief survey of these three components for becoming up to date. From, can be concluded that, Engineering (with > 60%) and Computer since (with 25%) are used more than the other areas. It can also means that automatic bridge inspection (ABI) is an attractive filed in the engineering domain.

In order to evaluate the relationship between the keywords and the abstract of the article, the evaluation, and text mining was conducted among the references. Using map and zoning software and research focus, research opportunities, and topics of the day in the field of automatic detection of concrete pavement cracking are extracted (see Fig. 3).

Fig. 3

Mapping of keywords and abstract for automatic cracking detection and classification using VOS viewer software tool

As shown in Figs. 3 and 4, it is clear that there is a close relationship between computing and engineering methods in order to manage the bridge failure assessment in recent years. So that more than 85% of publications are in this field. A general comparison was made using keywords in this regard. Based on this method, various keywords including Bridge + Inspection, Automatic + Bridge + Inspection, Cracking + Image + Processing, Concrete + Cracking + Detection, Concrete + Cracking + Classification, Concrete + Cracking + Quantification, Bridge + Assessment + Cracking, Remote + bridge + inspection in different domains were done according to the type of file and subject. In the field of bridge and surveillance, more than 43% of the papers journals articles, and over 50% of the papers are presented at conferences. Some of these documents, which are less than 4%, refer to overview articles. The statistics show that there is a little overview of the subject and the necessity of presenting this article. It is also a major part of the papers in the field of engineering and computer science (see Figs. 5, 6).

Fig. 4

The steps of the review task

Fig. 5

Comprehensive comparison of methods used by researchers in recent years based on keywords for a subject area and b type, for Bridge + Inspection, Automatic + Bridge + Inspection, Cracking + Image + Processing, Concrete + Cracking + Detection, Concrete + Cracking + Classification, Concrete

Fig. 6

Comprehensive comparison of methods used by researchers in recent years based on keywords for a subject area and b type for Concrete + Cracking + Quantification, Bridge + Assessment + Cracking, Remote + bridge + inspection

In the Automation section, surveys and evaluations of the bridge, the results of the survey show that most of the articles are with conference-based research, accounting for 54%, and about 40% of the research is published in articles in prestigious journals. Similar to the previous results, review articles are also deficient and less than 6%. Also, automation from both engineering and computer engineering has been used more than any other engineering department (85%) to automate the evaluation.

Cracking is one of the most visible and measurable distress in infrastructure. The most important signs of breakdowns on bridges are cracking. In this regard, more than 60% of the research published in journals and 34% of the conference papers specifically talked about it, and less than 4% of the articles reviewed the types of methods. Cracking is a major component of distress engineering, with 55% in the field of engineering and 31% in materials. In this case, computer engineering has developed a dynamic evolution in the field of automated evaluation over the past few years, using image processing techniques, artificial intelligence, classification, segmentation and engineering methods. The growth process of papers in the field of image processing and cracking in the automated evaluation of bridge and the technical building has been significant.

The first and most crucial assessment step is the distress diagnosis that reduces the time and cost of computation by separating the failure. Considering that there are several ways to address this issue, much research has been done in recent years, and more than 56% of articles in journals and about 40% in conferences have been felled. Review articles are also less than 4%. There is a higher degree of failure detection in the field of engineering and less than 6% of the papers in the field of computers. This suggests that in the field of automatic detection, despite significant growth in the years to come, there are excellent research opportunities for researchers (Table 1).

Table 1 Comparison of different documents for countries/territories

There are two general points of view regarding distress analysis, (1) the interpretation using accurate data, and decision-making based on the repeatable and accurate results that called objective, (2) the use of field methods that the results are not repeatable and the accuracy is somewhat acceptable that called subjective. In most of the commonly used methods for SHM and SHA, inspections and field tests are performed by experts to measure and analyze the damage condition, which is very costly, and time-consuming. Also, besides, field methods are subjective and dependent on personal diagnosis, which leads to a reduction in the accuracy and increase of human error. Every bridge distress based on routine bridge inspection evaluated and the treatment determined using a bridge management system (BMS) [19]. The cost and risk reductions in the inspection are always important for DOTs [19].

Various automatic and semi-automatic systems have width different technologies developed to help inspectors during inspections [20]. For example, Visual inspection [21], Unmanned Ground vehicles (UGVs) [22], Unmanned Aerial Vehicles (UAVs) [19, 23,24,25,26,27,28,29,30,31,32,33], Unmanned Marine Vehicles (UMVs) [34], Robotic under water [34], Operated and tele operated mobile robots [35], Ground penetrating radar (GPR) [36, 37], Climbing/Crawling robots [38, 39], Ultrasonic surface waves [40], The ROCIM system [35], Cable surface damage detection (CSDD) [41], RABIT autonomous bridge deck assessment tool [16], Sliding type bridge inspection robot [42, 43], Mobile robot in approach stage for a road bridge inspection [44], Parapet slider installed on a parapet [44], ER sensor equipped on the robotic system [45], UGV inspection platform used for automating data collection for visual inspection [22], DiddyBorg robotic platform [46], UAV bridge inspection robot and resulting infrared image [28], Small Unmanned Aircraft Systems (SUAS) instrument for bridge inspections [29], Multirotor UAS platform with multi-sensor equipment [32].

Although various hardware has been developed to evaluate and inspect components of bridges, the detection and classification of various cracks still do not perform well due to the complexity of multiple defect types of cracking. Accurate and fast classifying concrete distress during bridge inspection is still the subjective mission and remains as a problem [20]. Koch et al. [15] presents a comprehensive review on the computer vision-based defect detection and condition assessment and highlights the list of distress for bridges [15]. Based on their statement, they reviewed and discussed some of the existing algorithms for distress detection and classification [15]. Rose et al. [47] studied several crack detection and quantification algorithms for concrete bridges based on edge detection, segmentation, and classification. The paper; evaluates the challenges of automation; and finally stated that the supervised, semiautomatic approaches are effective methods toward automating the process.

Abdel-Qader et al. [48] reviewed numerous methods for edge detection and concluded the Wavelet method filter Haar is the most effective among filters for crack detection. However, they stated that noisy environment image analysis is questionable, and it needs a more intelligent method [48]. Recently, Yeum et al. [2] developed an automated region-of-interest localization and classification for vision-based visual assessment of civil infrastructure. Based on their method, complementary ROI is filtered using convolutional neural networks (CNN).

Hüthwohl et al. [20] developed a Multi-classifier for reinforced concrete bridge defects detection and classification. The methods can automatically multi-classify concrete bridge defects on image patches. They showed that their method could classify type distress 85% accuracy. Kim et al. [49] proposed a method for the classification of Crack/Noncrack from concrete surface images using machine learning. They stated that a serious challenge is to identify distress from an image containing cracks and crack-like objects, which are in concrete structures, prevalently.

In 2019, Li et al. [50] developed an automatic pixel-level multiple damage detection of concrete structure using Fully Convolutional Network (FCN). The results showed that the proposed model has 98.61% pixel accuracy, 91.59% mean pixel accuracy, 84.53% mean intersection over the union, and 97.34% frequency weighted intersection over the union. The results showed that the better performance of FCN for the detection of multiple concrete damages.

Recently in 2019, Ni et al. [51] proposed a Zernike-moment measurement of thin-crack width in images enabled based on dual-scale deep learning (DSDL). They proposed the novel crack width estimation algorithm based on the ZMO. Zhang et al. [52] developed a method for Concrete crack detection based on context-aware deep semantic segmentation network for various conditions.

Oliveira et al. [53] proposed an automatic mapping of cracking patterns on concrete surfaces using biological stains based on hyperspectral image processing (HSIP). The Super Cluster-Crack method (SC) is developed. A k-means clustering and grouping clusters are used for super clustering. Kong and Li [54] developed a method based on image overlapping for tiny crack extraction. Their method could classify crack from non-crack features. Dung and Anh, in 2019, developed an Autonomous concrete crack detection using the deep, fully convolutional neural network (CNN). Their method built on pre-trained networks with 97.8% accuracy, and 90% average precision [55].

A new FSM operator is proposed to search the double edges of cracks in image rows by Luo et al. [56]. They stated that the FSM operator has high sensitivity and the speed of the algorithm is worth six times faster than several typical algorithms, with an average processing time of megapixel images, 95 ms. Foreground/background separation technique for crack detection presented by Nayyeri et al., 2019. Based on their method, the firm edges extracted and edges distinguished from the local neighborhood. Then, based on the spatial distribution, the features extracted. Their methods showed an average f-measure of 75% in detecting the concrete cracks [57].

A new framework based on CNN, developed by Kim and Cho [58] for concrete structures crack assessment using the mask and region-based convolutional neural network (Mask R-CNN), under the title Image-based concrete crack assessment using mask and region. Based on their work. Yamaguchi et al. [59] used a high-speed crack detection method based on percolation image processing. It stated that the experimental result shows the efficiently reduces the computation cost.

Payab et al. [60], present a survey using image-based methods for crack analysis in the last two-decade (2002–2016), and a new method based on graph theory and image analysis used for concrete crack quantification. According to Zhong et al. [1], knowledge gaps in computer vision research in construction were discussed. Based on their research developments, knowledge gaps and implications for research are discussed. The co-occurrence network highlights a strong relationship between ‘image processing’ and ‘crack detection’ and ‘inspection’.

A Vision-based method for crack detection in steel structures using video feature tracking developed by Kong and Li [61]. Feature tracking method applied to the video for tracking of distress. Based on their work, because of the capacity of the camera resolution was one of the challenging issues that remain for future studies. Dorafshan et al. [62] conducted an investigating of the performance of six edge detectors for concrete crack detection. Also, they present a comprehensive comparison between the edge detectors and the DCNNs. A new hybrid crack detection method proposed by a combination of the DCNN and the edge detector. They concluded that the hybrid method had less noise than the other methods. Fractal and multifractal characteristics of crack as features can be used in crack detection an quantification [63].

A novel deep learning technique named a fully convolutional network (FCN) developed by Yang et al. (2018) for Automatic Pixel-Level Crack Detection and Measurement [64]. Kim et al. [10], proposed an application of crack identification method for inspection of aging concrete bridge based on image processing and UAVs. The cracks on the bridges surface detected based on convolutional neural networks (R-CNN). Zakeri et al. [6] conducted a comprehensive review of a wide range of image-based techniques for crack detection, classification, and quantification. Hüthwohl et al. [20] present an approach that can classify multiple defect types with an average mean score of 85%. Base on their work, a multi-classifier system assigns none, one, or multiple defect labels to an image containing potentially weak areas.

Most of the researches focused on cracks/non-crack, as an essential factor in detection. In conclusion, in the last decades, many image-based methods have been proposed for automatic crack detection, classification, and quantification [6, 12].

However, thesis approaches faced new problems related to control the image interpretation process. Make an optimum decision in unknown situations, make progress from insufficient knowledge that extracted from the learning step and working under different environments are several challenging problems related to existing methods for distress detection and classification. One of the most important weaknesses of most presented methods in literature is that these techniques do not come across the above robustness necessities. In addition, these studies show that existing methods are not capable of self-learning, and influencing the environment.

On the other pole, multi-agent image interpretations have several advantages abilities such as using various domains for knowledge representation, employing a width range of image processing methods and learning algorithms for detection and classification, online and parallel rule maintenances and tuning. Therefore, in this article, we aim to present a method for identifying and classifying cracks using an intelligent multi-agent system. In this regards various multi-agent image interpretation systems have been reported in the literature [65,66,67,68,69,70,71,72,73]. These systems combine different methods for the preprocessing of images and operate in environments with uncontrolled conditions such as brightness, contrast, or sharpness [65]. A set of agents in a shared environment, which learn from an intelligent method, make an intelligent multi-agent [74].

1.1 Gaps in the Knowledge and Research Questions

Hence, in related works does not exist an adaptive procedure for detection and classification of distress on bridges elements. The previous methods work only for simple geometrical crack and limited environmental conditions; however, in the real world the geometrical is complicated and environmental conditions suffer from clearness. To date, no research has been presented to identify and classify complex cracking patterns in real-world situations. Cracks in bridges have a complex nature and are unable to adapt to real conditions such as shadows, external issues, and sulfated environments to analysis. Therefore, a new approach is needed to handle the complexity pattern ability to work under vague conditions to classify potentially distress form complex non-distress texture. In this research, a new intelligent method based on a company of agents developed for solving this problem and classifying distress (individual, pattern, and random cracking) that has high accuracy, sensitivity and, speed. The method presented in this paper is an objective automatic image-based system for crack detection and type classification in concrete elements. These systems do not have the weaknesses of field methods and they are highly accurate in comparison with other objective methods and provide satisfactory results.

2 Methodology

The main objective of this study is to provide a new method for identifying and classifying different types of cracking. The automatic distress detection and classification without human intervention and the least error and in the shortest time as mentioned in Sect. 1.1, is the main objective of this research. Due to the complexity of the crack structure and the unique behavior of each crack and consequently, the wide variation in the appearance of the cracks, the combination of fuzzy inference systems have been used. The fuzzy inference system, play a role as a set of intelligent agents in a multi-agent system that interacts and shares information with others. One of the most critical benefits of the fuzzy logic system is its readability and ease of extensibility.

An important step in designing a fuzzy classification system is to optimize the parameters. In this study, coefficients of membership functions are considered as variables whose optimal values are obtained using training and testing data sets based on an optimization method. There are various methods available for optimization, which are divided into two general categories of exact and approximate methods [75, 76].

The exact methods are capable of finding the optimal solution accurately, but they don’t work well on complex optimization problems [77], and their time of tuning in these issues increases exponentially. The second category or approximate methods are divided into three main groups of heuristic, meta-heuristic and hyper-heuristic algorithms [78]. This category includes many algorithms that can find good (near to optimal) answers for complex problems in a short time. In this study, the particle swarm optimization [79] method has been used to optimize the coefficients of fuzzy membership functions. The PSO algorithm is a population-based meta-heuristic method that exploits swarm intelligence. Benefit from memory, collaboration and sharing of information between particles, high convergence rate, better flexibility versus optimal local problem and easy to implement and run are the reasons for choosing the PSO method to optimize fuzzy membership functions in this study. In the following, the design of fuzzy classification systems, multi-agent systems and their specifications, and the particle swarm optimization are presented.

2.1 Fuzzy Classification System

The design of a fuzzy classification system in this study consists of five steps:

• Step 1 Collecting database from all classes and dividing them into two categories of training and testing data sets: After determining the number of classes, 70% of the database is selected randomly and without displacement as a training set, and the remaining 30% is considered as a testing set. The selection of the training set should be such that there is at least one data from each class, otherwise,; the choice must be resumed.

• Step 2 Extracting the essential features of the training set and determining their range: Some features are selected based on the type of problem and are calculated for the training data set. Then the range of these features is determined by their minimum and maximum values.

• Step 3 Generating the basic fuzzy classification system based on the obtained features: The extracted features in the second step with their limits are the input of the basic fuzzy classifier that is produced at this stage and the system’s output specifies the class type (crack or non-crack). The rules used in the central engine, are designed, based on fuzzy theory knowledge extracted from the data set and available theoretical and empirical research. Then the fuzzy classification modules designed, and the tuning of the parameters set and firing rule are down based on the error of testing data set evaluation.

• Step 4 In this step the parameters of membership function are Optimized by using an optimization method, PSO on the training data: The most adaptive fuzzy inference system is achieved by optimizing the membership function coefficients using particle swarm optimization (PSO) method (in this study). The membership function in each level consists of four coefficients, assuming that all membership functions are vertical trapezoidal type. In this way, we have a total of 8 coefficients for each input or output in the fuzzy rule set, taking into account two low and high levels for all membership functions. Four of these coefficients are unknown that should be optimized and the others are equal to the minimum and maximum values of the feature (see Fig. 7).

  Fig. 7

  

  (2-levels) Trapezoidal fuzzy membership function

• Step 5 dedicated to testing the best fuzzy classifier obtained from training data: In the last stage, the optimized fuzzy system in the step 4 is used to test the data. Then the reliability of the optimized fuzzy classification system is evaluated through accuracy and error.

2.2 Agent-Based Modelling

An Intelligent Agent (IA) denotes to an autonomous object with the special duty to move and act towards goals, under the condition of an environment based on information received from sensors and deduct based on an intelligent knowledge base. The IA may also use knowledge to move toward the goal. The AI can be designed very simple or very complex. When an individual agent can’t solve a high complexity problem, Multi-agent systems can be used to handle the complexity. The main goal of an agent-based model is to search for analytical insight into the collective behavior of agents obeying simple rules, typically in natural systems, rather than in solving specific practical or engineering problems. Based on Russell and Norvig [80], agents classified into several groups based on their intelligence capability. The agent used in this work is Learning agents that it allows the agents to operate in unknown environments (bridge image) and to become more competent than its basic knowledge.

A multi-agent system can be a reasonable frame for an image interpretation system as long as an intelligent learning method in AI. Architecture In this paper have chosen based on agent and fuzzy rules for constructing general intelligent learning systems, which has been tested successfully on many distress and cracks databases. Fuzzy reasoning is non-deterministic which allows it to withdraw previous conclusions and thus to learn from knowledge. A global view of the multi-agent system design is given in Fig. 8.

Fig. 8

A general multi-agent system architecture

2.3 Particle Swarm Optimization (PSO)

As mentioned in Sect. 2.1, the Particle Swarm Optimization (PSO) is used to optimize the fuzzy membership function coefficients and to obtain the most adaptive fuzzy classification system based on the behavior of a set of particles [81, 82]. Each particle represents a fuzzy inference system, and the last selected particle represents the optimized fuzzy classification system [83, 84]. The flowchart of the PSO algorithm is shown in Fig. 10 and the algorithm of this method can be summarized in four steps as follows:

• 1. Determine the specification and generate the initial population of particles:

The input parameters of this algorithm are the number of particles (n), the maximum number of iterations $({\text{Max}}_{\text{ite}} )$, the constants of the fitness function ($\sigma_{\text{e}}$ and $\sigma_{\text{r}}$), the constants of the PSO method (d, c₁ and c₂) and the minimum and maximum inertia weights (${\text{w}}_{\hbox{min} }$ and ${\text{w}}_{\hbox{max} }$) which should all be adjusted. Also, the total number of inputs and outputs of the rules is determined without repetition. The initial population of the particles (P) is generated randomly by the Adaptive network-based fuzzy inference system (ANFIS) using the following equations:

$$\begin{aligned} & {\text{P}} = [\underline{\text{P}}_{1} ,\underline{\text{P}}_{2} ,\ldots,\underline{\text{P}}_{\text{h}} ,\ldots,\underline{\text{P}}_{\text{n}} ] \\ & \underline{\text{P}}_{\text{h}} = [{\text{X}}_{1}^{{1{\text{h}}}} ,{\text{X}}_{1}^{{2{\text{h}}}} , \ldots ,{\text{X}}_{1}^{{{\text{m}}_{1} {\text{h}}}} ,{\text{X}}_{2}^{{1{\text{h}}}} ,{\text{X}}_{2}^{{2{\text{h}}}} , \ldots ,{\text{X}}_{2}^{{{\text{m}}_{2} {\text{h}}}} ,\ldots,{\text{X}}_{\text{k}}^{{1{\text{h}}}} ,{\text{X}}_{\text{k}}^{{2{\text{h}}}} , \ldots ,{\text{X}}_{\text{k}}^{{{\text{m}}_{\text{k}} {\text{h}}}} ] \\ & \underline{\text{V}}_{\text{h}} = [\mu_{1}^{{1{\text{h}}}} ,\mu_{1}^{{2{\text{h}}}} , \ldots ,\mu_{1}^{{{\text{m}}_{1} {\text{h}}}} ,\mu_{2}^{{1{\text{h}}}} ,\mu_{2}^{{2{\text{h}}}} , \ldots ,\mu_{2}^{{{\text{m}}_{2} {\text{h}}}} ,\ldots,\mu_{\text{k}}^{{1{\text{h}}}} ,\mu_{\text{k}}^{{2{\text{h}}}} , \ldots ,\mu_{\text{k}}^{{{\text{m}}_{\text{k}} {\text{h}}}} ] \\ & {\text{X}}_{\text{j}}^{\text{ih}} = [{\text{x}}_{{3,{\text{j}}}}^{\text{ih}} ,{\text{x}}_{{4,{\text{j}}}}^{\text{ih}} ,{\text{x}}_{{5,{\text{j}}}}^{\text{ih}} ,{\text{x}}_{{6,{\text{j}}}}^{\text{ih}} ] \\ \\ & {\text{h}} = \{ 1,2, \ldots ,{\text{n}}\} \\ & {\text{i}} = \{ 1,2,\ldots,{\text{m}}_{\text{j}} \} \\ & {\text{j}} = \{ 1,2,\ldots,{\text{k}}\} \\ \end{aligned}$$

(1)

where $\underline{\text{P}}_{\rm h}$ is the position of the particle h (from n particles), k is the number of rules and m_j is the total of the inputs and outputs of j-th rule. ${\text{X}}_{\text{j}}^{\text{ih}}$ is the i-th feature of the k-th fuzzy rule which includes four variables ${\text{x}}_{{ 3 , {\text{j}}}}^{\text{ih}}$ to ${\text{x}}_{{ 6 , {\text{j}}}}^{\text{ih}}$ (Fig. 9). Each variable is derived based on its minimum and maximum values by Eq. (2).

$$\begin{aligned} & {\text{x}}_{\text{s,j}}^{\text{ih}} = {\text{x}}_{\text{s,j}}^{\text{ih,min}} + ({\text{x}}_{\text{s,j}}^{\text{ih,max}} - {\text{x}}_{\text{s,j}}^{\text{ih,min}} ).{\text{rand}}( \, ) \\ & {\text{s}} = \{ 3,4,5,6\} \\ \end{aligned}$$

(2)

Fig. 9

Particle size description in PSO method

Then the initial velocity vector of particle h ($\underline{\text{V}}_{\rm h}$) is obtained as follows:

$$\begin{aligned} & \underline{\text{V}}_{\text{h}} = [\mu_{1}^{{1{\text{h}}}} ,\mu_{1}^{{2{\text{h}}}} , \ldots ,\mu_{1}^{{{\text{m}}_{1} {\text{h}}}} ,\mu_{2}^{{1{\text{h}}}} ,\mu_{2}^{{2{\text{h}}}} , \ldots ,\mu_{2}^{{{\text{m}}_{2} {\text{h}}}} ,\ldots,\mu_{\text{k}}^{{1{\text{h}}}} ,\mu_{\text{k}}^{{2{\text{h}}}} , \ldots ,\mu_{\text{k}}^{{{\text{m}}_{\text{k}} {\text{h}}}} ] \\ & \mu_{\text{j}}^{\text{ih}} = [\alpha_{{3,{\text{j}}}}^{\text{ih}} ,\alpha_{{4,{\text{j}}}}^{\text{ih}} ,\alpha_{{5,{\text{j}}}}^{\text{ih}} ,\alpha_{{6,{\text{j}}}}^{\text{ih}} ] \\ & \alpha_{\text{s,j}}^{\text{ih}} = \frac{1}{20}\left( {{\text{x}}_{\text{s,j}}^{\text{ih,max}} - {\text{x}}_{\text{s,j}}^{\text{ih,min}} } \right)).{\text{rand}}( \, ) \\ \end{aligned}$$

(3)

• 2. Calculate the Fitness function and set the initial best position of each particle and the global best position in the particle population from the beginning:

Assuming that the number and combination of the rules (${\text{r}}_{\text{h}}$) are constant and do not change, the fitness function is considered as a function of the number of false classified patterns (${\text{NICP}}$) as Eq. (4). The higher the fitness function, the more satisfying the performance of the fuzzy system will be. $\sigma_{\text{e}}$ and $\sigma_{\text{r}}$ are constants.

$$\begin{aligned} & {\text{f}}_{\text{h}} = {\text{fit}}(\underline{\text{P}}_{\text{h}} ) \\ & {\text{f}}_{\text{h}} = exp\left( { - \frac{{{\text{NICP}}\left( {\underline{\text{P}}_{\text{h}} } \right)}}{{\sigma_{\text{e}} }}} \right) \, \cdot exp\left( { - \frac{{{\text{r}}_{\text{h}} }}{{\sigma_{\text{r}} }}} \right) \\ \end{aligned}$$

(4)

Initially, the best particle position ($\underline{\text{P}}_{\text{h}}^{\text{PBest}}$) is equal to the initial particle position and the best value of the fitness particle function (${\text{f}}_{\text{h}}^{\text{PBest}}$) is equal to the initial value of the particle’s fitness:

$$\begin{aligned} \underline{\text{P}}_{\text{h}}^{\text{PBest}} = \underline{\text{P}}_{\text{h}} \hfill \\ {\text{f}}_{\text{h}}^{\text{PBest}} = {\text{f}}_{\text{h}} \hfill \\ \end{aligned}$$

(5)

As seen in Eq. (6), the particle with the highest value of the Fitness function (particle number h_Best), is considered to be the particle with the best position among the whole initial population ($\underline{\text{P}}_{\text{h}}^{\text{GBest}}$). The fitness function of this particle is considered as the best fitness function in the particle population (${\text{f}}_{\text{h}}^{\text{GBest}}$):

$$\begin{aligned} & {\text{h}}\_{\text{Best}} = { \arg }\mathop { \hbox{max} }\limits_{{{\text{h}} = 1}}^{\text{n}} ({\text{f}}_{\text{h}} ) \\ & \underline{\text{P}}_{\text{h}}^{\text{GBest}} = \underline{\text{P}}_{{{\text{h}}\_{\text{Best}}}} \\ & {\text{f}}_{\text{h}}^{\text{GBest}} = {\text{f}}_{{{\text{h}}\_{\text{Best}}}} \\ & {\text{ite}} = 1 \\ \end{aligned}$$

(6)

• 3. Update the problem parameters:

In this step, the velocity vector and particle position are updated by the Eqs. (7) and (8):

$$\begin{aligned} & \underline{\text{V}}_{\text{h}} = {\text{w}} \cdot \underline{\text{V}}_{\text{h}} + {\text{c}}_{1} \cdot {\text{rand}}( \, ) \cdot (\underline{\text{P}}_{\text{h}}^{\text{GBest}} - \underline{\text{P}}_{\text{h}} ) + {\text{c}}_{2} \cdot {\text{rand}}( \, ) \cdot (\underline{\text{P}}_{\text{h}}^{\text{PBest}} - \underline{\text{P}}_{\text{h}} ) \\ & {\text{w}} = {\text{w}}_{ \hbox{max} } - ({\text{w}}_{ \hbox{max} } - {\text{w}}_{ \hbox{min} } ) \cdot {{\text{ite}} \mathord{\left/ {\vphantom {{\text{ite}} {{\text{Max}}_{\text{ite}} }}} \right. \kern-0pt} {{\text{Max}}_{\text{ite}} }} \\ \end{aligned}$$

(7)

and

$$\underline{\text{P}}_{\text{h}} = \underline{\text{P}}_{\text{h}} + \underline{\text{V}}_{\text{h}}$$

(8)

where w is the inertial coefficient obtained based on the maximum and minimum weight of the inertia. ${\text{c}}_{1}$ and ${\text{c}}_{2}$ are the acceleration coefficients, the values of which are between 0 and 1. The velocity vector is reduced by decreasing coefficient d (Eq. (9)).

$$\underline{\text{V}}_{\text{h}} = \underline{\text{V}}_{\text{h}} \cdot {\text{d}};\quad {\text{d}} \in [0,1]$$

(9)

The new value of the fitness function of particles is obtained from Eq. (4). The best position of each particle and the global best position in the particle population are updated by the Eq. (10).

$$\begin{aligned} & {\text{if}}\;{\text{ f}}_{\text{h}} > {\text{f}}_{\text{h}}^{\text{PBest}} \Rightarrow \, \\ & {\text{f}}_{\text{h}}^{\text{PBest}} = {\text{f}}_{\text{h}} \, , \, \underline{\text{P}}_{\text{h}}^{\text{PBest}} = \underline{\text{P}}_{\text{h}} \\ \\ & {\text{if}}\;{\text{ f}}_{\text{h}}^{\text{PBest}} > {\text{f}}_{\text{h}}^{\text{GBest}} \Rightarrow \, \\ & {\text{f}}_{\text{h}}^{\text{GBest}} = {\text{f}}_{\text{h}}^{\text{Pbest}} \, , \, \underline{\text{P}}_{\text{h}}^{\text{GBest}} = \underline{\text{P}}_{\text{h}}^{\text{PBest}} \\ & {\text{ite}} = {\text{ite}} + 1 \\ \end{aligned}$$

(10)

• 4. Criteria to control the stop conditions and repeat step 3 to satisfy one of these conditions:

The stop condition of the PSO algorithm is to reach the maximum number of iterations or to reach a threshold fitness function, or several iterations, in which fitness function does not vary much (tolerance is a small number). Otherwise, Step 3 should be repeated until one of the stop conditions is fulfilled:

$${\text{or}}\;\left\{\begin{array}{*{20}l} {{\text{ite}} \geq \max_{\text{ite}} } \hfill \\ {\text{f}}_{\text{h}}^{\text{Gbest}} = {\text{fitness}}_{ \max} \hfill \\ {\text{tolerance}} \leq 10^{ - 12} \hfill \\ \end{array} \right. \Rightarrow \quad STOP\;Algorithm$$

(11)

After the stop condition is established, the particle with the highest fitness function is selected as the optimized fuzzy classification system (Fig. 10).

Fig. 10

Basic particle swarm optimization flowchart

3 Presented Method

The main purpose of this paper is to detect the crack distress from non-cracked elements, as well as classify various types of cracking in the images. The presented methods to achieve these questions are discussed as follows sections. The first section presents designed anent based system for crack detection that classifies input images into two groups of crack and non-crack. The images selected from the first section are used as an input for type classification using section two that is described in the second section. This system classifies the input images into three types of individual, pattern, and random. Due to the vagueness inherent of cracking aspect of wide, rotation, pattern, connectivity, roots, main and secondary cracks, the complexity of their pattern is high, therefore for handling this ambiguous, the combination of fuzzy inference systems has been used as intelligence in the center of a multi-agent system. The particle swarm optimization (PSO) method employed for the optimization of these two systems (see Fig. 11).

Fig. 11

The multi-agent architectures for presented crack detection and classification systems

In this paper, we used the Soar Cognitive Architecture (SCA) to make a multi-agent intelligent system. Soar is a cognitive architecture that made from long-term and short-term memories and used for inference [85]. This architecture, which has been successfully tested on various artificial intelligence issues in recent decades [86,87,88,89,90,91,92,93,94], has a mini-mechanism that enables it to operate under unknown circumstances. In this architecture, each agent is responsible for identifying only one type of object in the image, and all agents solve the problem in parallel.

3.1 Crack Recognition Using Fuzzy Agent-Based System

In this paper, in order to detect cracks from non-cracks, two main features, “Shape” and “Spalling”, are defined, each one is evaluated using an index. In fact, the probability that an object is a crack is assessed based on these features. The Shape feature represents the object in terms of its appearance and is defined as a function of the two parameters, Compactness and LAP. Each of these parameters has its own threshold (values 1 and 2 in Fig. 12). The second feature, Spalling, is a benchmark for defining the amount of object peeling and separate the crack from the spall. According to references and protocols, spalling is defined as a type of concrete defects that is different in nature with the crack and should be distinguished. This feature is defined based on the parameters Area_Spl and Length_Spl that have their special threshold (value 3 and value 4 in Fig. 12). The Decision Tree (DT) for describing crack recognition illustrated in Fig. 12.

Fig. 12

An ontology scheme for describing the apparent nature of the cracks

3.1.1 Development of a Crack Recognition Fuzzy Model

According to the definition presented for the nature of the crack, the fuzzy detection system of the crack is designed as a hybrid fuzzy system shown in Fig. 13. The system has two inputs, Area_Spl and Length_Spl, each of them is the output of an independent fuzzy inference system. The details of designing these fuzzy systems (Shape FIS, Spalling FIS and crack detection FIS) are described below.

Fig. 13

Presented hybrid FIS model for crack detection system (4 inputs, 1 output, 12 rules)

• I. Shape FIS

As mentioned, and shown in Fig. 13, the Shape feature is defined as a function of two computational parameters Compactness and LAP, derived from Eq. (12).

$$\begin{aligned} & {\text{Compactness}} = \frac{{{\text{P}}_{\text{circle}} }}{{{\text{P}}_{\text{obj}} }} \\ & {\text{LAP}} = \left( {\frac{{{\text{L}}_{\text{skl}} \cdot {\text{A}}_{\text{obj}} }}{{{\text{P}}_{\text{obj}} }}} \right)^{0.1} \\ \end{aligned}$$

(12)

where ${\text{P}}_{\text{circle}}$ represents the perimeter of a circle with the same object’s area. ${\text{P}}_{\text{obj}}$ and ${\text{A}}_{\text{obj}}$ are, respectively the object’s perimeter and area, and ${\text{L}}_{\text{skl}}$ is the total length of the object’s skeleton.

• II. Spalling FIS

The feature of the Spalling is extracted based on the two computable parameters of Area-SPL and Length-SPL according to the Eq. (13) (Fig. 13). It is assumed that the spalling occurs in areas where the thickness is more than 20% of the total length of object. The distress cannot be a crack if the amount of spalling exceeds its limit of extent, and it will be a spall failure that is not the subject of this article.

$$\begin{aligned} & {\text{Length}}\_{\text{SPL}} = \frac{{2 \times {\text{Length}}\, \, ({\text{D}} > 0.2{\text{Length}}_{\text{all}} )}}{{{\text{Length}}_{\text{all}}^{1.5} }} \\ & {\text{Area}}\_{\text{SPL}} = \frac{{50 \times {\text{Area}}\, \, ({\text{D}} > 0.2{\text{Length}}_{\text{all}} )}}{{{\text{Area}}_{\text{all}}^{2} }} \\ \end{aligned}$$

(13)

where ${\text{Length}}\, \, ({\text{D}} > 0.2{\text{Length}}_{\text{all}} )$ and ${\text{Area}}\, \, ({\text{D}} > 0.2{\text{Length}}_{\text{all}} )$ in these equations, represent the length and area of the parts of the object that are thicker than 0.2 times the total length of object. ${\text{Length}}_{\text{all}}$ and ${\text{Area}}_{\text{all}}$ are the length and area of the entire object, respectively.

• III. Crack detection FIS

As illustrated in Fig. 13, the Crack Detection System has two inputs, each of that is the output of a separate fuzzy system that was described in the previous II. The output of the fuzzy crack detection system is given as a percentage index (fuzzy), but in order to detect the cracks from non-cracks definitively, it is necessary to express the results as definitive (Crisp). For this purpose, the threshold of 50% is considered to split the results into two classes of cracks and non-cracks and give a definitive diagnosis. In this way, if the output of crack detection FIS exceeds 50%, it indicates positive detection (or crack), and less than 50% indicates a negative detection (or non-crack). In the rule base of the fuzzy detection system, the 12 rules are defined, that the knowledge base extracted is presented in Table 2.

Table 2 Fuzzy rule base in crack detection system

3.1.2 The Optimization of Fuzzy Membership Functions

In the presented intelligent MAS, a set of rules is designed using membership functions coefficients that the inputs and outputs of the fuzzy systems are considered as variables. The optimization of these coefficients is performed using the PSO method, as described in Sect. 2.3.

In order to optimize the membership function coefficients of the crack detection system, the number of particles (population size) in the PSO method is considered to be 100 particles. The total number of rules and the total number of inputs and outputs (without repetition) are 12 and 7 respectively. Given that 4 of the total coefficients of each membership function are unknown, the sum of all variables is (6 + 1) × 4=28 (Fig. 9), whose optimal values are obtained by the PSO method. Parameters used in this method are defined according to Table 3 and the optimization results are presented in Sect. 5.1.

Table 3 Parameters used for optimization of crack detection system in PSO method

3.2 A Fuzzy Classifier for Crack Classification

According to the protocols [95] and standards [96, 97], The cracks are classified into two general categories of individual and non-individual cracks, based on their patterns [98] (see Fig. 14). Individual crack is a linear crack, which classified into three types of longitudinal, transverse, and diagonal cracks [49] according to the orientation of their original skeleton relative to the longitudinal direction of the distress of the member. The second category, non-individual cracks, consists of two types of pattern and random cracks. Pattern or Map crack is a network of connected random cracks on the surface of concrete, which are usually shallow and superficial (depth less than 1.8 in.) and it is not structural distress. The area enclosed by these cracks is usually between 0.5 and 4 in. in square [49]. Random or multiple cracks are a set of individual cracks that have been created together at different angles. This type of cracks may have partial meshes, but they do not have a regular pattern.

Fig. 14

Classification of cracking types in terms of appearance

Since the determination of the type of individual cracks (longitudinal, transversal and diagonal) is possible based on the crack’s direction and has been investigated in the previous studies [49], this paper only focuses on detection and classification the individuality of the crack. So, in the design of the presented system, three types of individual, pattern, and random cracks have been investigated. The four main parameters introduced in this system for detection and classification of cracks are Mesh, Lengthy, Multiplicity, and Deviation, which the conditions of these features are shown in Fig. 15.

Fig. 15

An ontology scheme for describing crack types and properties

3.2.1 Development of a Crack Type Classification Fuzzy Model

In designing the fuzzy classification system, a combination set of five fuzzy logic systems is used, in which the output of the first four fuzzy systems is given as input to the main fuzzy classifier system (Fig. 16). The first four fuzzy inference systems are the Mesh FIS, Lengthy FIS, Multiplicity FIS, and Deviation FIS. In the following, the details of the design of these four fuzzy systems and the classifier FIS are presented.

Fig. 16

Presented hybrid FIS model for crack type detection system (7 inputs, 3 output, 20 rules)

• I. Mesh FIS

The first feature in the fuzzy classifier is called “Mesh”, which represents the degree of gridness, and uses two input parameters of the Intension and Extent in its fuzzy logic system, which are described in Eq. (15) (see Fig. 16).

$$\begin{aligned} & {\text{Intension}} = \frac{{{ \ln }\,({\text{CycleNum}})}}{{{ \ln }\,({\text{maxCycleNum}})}} \\ & {\text{Extent}} = \frac{{{ \ln }\,({\text{holeArea}})}}{{{ \ln }\,({\text{convxArea}})}} \\ \end{aligned}$$

(14)

where, holeArea and convxArea are respectively the total areas of the meshes (named cycles) and the area of the convex hull around the crack. CycleNum is the number of existing cycles and maxCycleNum is the maximum number of possible cycles obtained from the Eq. (16), assuming k is the number of objects (see Fig. 17 as an example).

$${\text{maxCycleNum}} = \left\{ {\begin{array}{*{20}l} 0 \hfill & {{\text{k}} < 6} \hfill \\ 1 \hfill & {{\text{k}} = 6} \hfill \\ 2 \hfill & {{\text{k}} = 7} \hfill \\ {\begin{array}{*{20}l} {{{({\text{k}} - 1)} \mathord{\left/ {\vphantom {{({\text{k}} - 1)} 2}} \right. \kern-0pt} 2}} \hfill & {k,odd} \hfill \\ {{\text{k}}/2} \hfill & {k,even} \hfill \\ \end{array} } \hfill & {{\text{k}} > 7} \hfill \\ \end{array} } \right.$$

(15)

Fig. 17

Calculate CycleNum and maxCycleNum for a crack image sample. a Crack skeleton image; b crack separation into its components (in this example, crack consisting of 24 objects and 19 points); c The graph related to b includes 6 cycles (CycleNum = 6); d the graph consisting of 24 objects related to the state with the largest number of possible cycles (has 12 cycles (maxCycleNum = 12) according to Eq. (15))

• II. Lengthy FIS

The second determinative feature in cracks classification is the “Lengthy” that indicates the degree of linearity and determined by a vector of feature consists of three input parameters compactness_fill, LAP_fill and Cvar_fill in the Lengthy FIS (see Fig. 16). The first two parameters are calculated as described in Sect. 3.1.1. As for where the meshes crack, filled, in this case. The Cvar_fill parameter is the circular variance of the crack in the filled state. The circular variance can be calculated by the Eq. (17).

$$\begin{aligned} & {\text{Cvar}} = \frac{1}{{{\text{N}}\mu_{\text{r}}^{2} }}\sum \left(\left\| {{\text{P}}_{\text{i}} - } \right.\left. \mu \right\| - \mu_{\text{r}}^{2} \right);\quad {\text{Cvar}} \ge 0 \\ & \mu = \frac{1}{\text{N}}\sum {\text{P}}_{\text{i}} \\ & \mu_{\text{r}} = \frac{1}{\text{N}}\sum \left\| {{\text{P}}_{\text{i}} - } \right.\left. \mu \right\| \\ \end{aligned}$$

(16)

where ${\text{P}}_{\text{i}}$ is the position on the shape’s contour $\mu$ and $\mu_{\text{r}}$ represent the geometric center of the shape and its mean radius, respectively.

• III. Multiplicity FIS

The number of cracks is the third most important feature in the crack type detection system, which is based on just one input, the Ngroups (Fig. 16). Ngroups is the number of cracks in a set after applying the Merging Algorithm designed in this study. By applying this algorithm, cracks with skeletal length less than the specified value that exist in the neighborhood of each other (if there are) are merged together and considered as a single set. This algorithm is applied to identify random cracks, which are irregular sets of cracks in each other’s neighbors (see Fig. 38). If this algorithm is not applied, the cracks in these sets of cracks incorrectly classified independently as individual cracks.

• IV. Deviation FIS

The last feature introduced to detect crack type is “Deviation”, which represents the deviation of cracks directions from each other in a set of cracks (includes more than one crack). This feature is also considered to enhance the accuracy of random crack detection. The only input of the Deviation FIS is the coefficient of variation of cracks directions in the set (CV), which is calculated from the Eq. (17).

$${\text{CV}} = \frac{\sigma }{\text{M}};\quad \sigma = \left( {\frac{1}{\text{n}}\sum \sum\limits_{{{\text{i}} = 1}}^{\text{n}} {(\theta_{\text{i}} - } {\text{M}})^{2} } \right){}^{{{\raise0.7ex\hbox{$1$} \!\mathord{\left/ {\vphantom {1 2}}\right.\kern-0pt} \!\lower0.7ex\hbox{$2$}}}};\quad {\text{M}} = \frac{1}{\text{n}}\sum \sum\limits_{{{\text{i}} = 1}}^{\text{n}} {\theta_{\text{i}} }$$

(17)

where $\sigma$ and M are the standard deviation and the mean of cracks directions relative to the horizon line in the crack set, respectively. N is the number of cracks in the set and $\theta_{\text{i}}$ is the direction of each crack relative to the horizon line. If a crack is separate and not a member of the set, the CV value is considered to be zero.

• V. Type detection FIS

As mentioned in section IV, and shown in Fig. 16, the crack type classifier consists of four inputs, each of which is an output of a fuzzy system. The classifier produces three outputs which are expressed as percentages of rule firing. These three outputs represent the probability that the investigated cracks are in each of the three classes of cracks, consists of Individual, Pattern, and Random.

The knowledge base made of several extracted rule that includes 20 rules. The description of these rules, along with the rules of Mesh, Lengthy, Multiplicity, and Deviation FIS’s, are presented separately in Table 4.

Table 4 Fuzzy rule base in crack classification system

3.2.2 The Optimization of Fuzzy Membership Functions

As discussed in Sect. 2.3, the PSO method is selected to optimize the method for type classification. The coefficients of the membership functions of the inputs and outputs are optimized using PSO. Table 5 shows the input parameters of the particle swarm optimization algorithm. As shown in Fig. 16, the total number of inputs and outputs (without repetition) of the system is equal to 14. According to Fig. 9 and the description in Sect. 3.1.2, 56 variables are optimized, generally.

Table 5 Parameters used for optimization of crack classification system in PSO method

4 Crack Retrieval Method

In the pre-processing phase, image enhancement algorithms are used to reduce noise, which results in the loss of some image information in low-intensity crack areas. In this paper, a method for recovering and connecting discrete cracks is presented to improve this situation. The basis of this method is to connect the parts closely together in one direction. The inputs contain two parameters SS and min_Angle, representing the dimension of the neighborhood square matrix and the least acceptable angle to be in the same orientation, respectively. An example is presented in Figs. 18, 19 and 20 to explain this method and the algorithm is designed in 9 steps as follows:

Fig. 18

An example to explain steps 1 and 2 in crack retrieval method. a A binary image containing some cracks; b endpoints shown as red points on cracks skeleton; c extracted the SS * SS neighborhood square matrix to the center of an endpoint. (Color figure online)

Fig. 19

Applying steps 3–7 of the crack retrieval algorithm on Fig. 18c. a, b Connecting the central endpoint (red points) to other neighboring endpoints (green points) and calculating the angles between the crack direction and connection lines; c removing those connections that intersect with the original image (blue cross) or their angle to the crack direction is less than “min_Angle” (red cross); d selecting the connection that has the highest value from Eq. (18). (Color figure online)

Fig. 20

The output of the crack retrieval algorithm for Fig. 18a. a Cracks skeleton after applying the retrieval method for all endpoints of the image; b retrieved cracks with real thickness

• Step 1 Extract the entire endpoints in the image after extracting the cracks skeleton and removing the small objects.

• Step 2 Extract the SS * SS neighborhood square matrix to the center of each endpoint in the main image.

• Step 3 Identify other endpoints in the neighborhood square.

• Step 4 Calculate the angles between the crack direction (based on the connection line between the central endpoint and the second pixel connected to it) and the connection lines between central endpoint with other neighboring endpoints.

• Step 5 Connect the central endpoint to other neighboring endpoints by the shortest direct path.

• Step 6 Remove connections that intersect with the pixels of the original image or their angle to the crack direction (obtained from step 4) is less than the specified value “min_Angle”.

• Step 7 Select the connection that has the highest value for the following ratio:

  $$\alpha ({\text{i}}) = \frac{{{\text{tetha}}({\text{i}})}}{{{\text{dst}}({\text{i}})}}$$

  (18)

  where ${\text{tetha}}({\text{i}})$ is the angle between the new connection and the crack direction (between 0 to π), and ${\text{dst}}({\text{i}})$ is the distance between each endpoint and the central endpoint.

• Step 8 Steps 2 through 7 are repeated for all image endpoints. The connection value is set to one for each selected connection in step 7. This produces a N * N matrix where N is the total number of image endpoints. In this matrix, the number 1 represents the connection and zero represents the non-connection between two endpoints.

• Step 9 The symmetric components in the obtained matrix are retained unchanged and the rest are considered zero. In this way, only the bilateral connections that are will remain and the rest will be removed. Finally, in the output image, connections are made between the endpoints aligned in one direction and are in close to each other.

To more illustrate the effect of the presented crack retrieval method, an example of real RGB crack image is provided. Figures 21 and 22 are related to the identification of discontinuous crack segments before and after applying this method, respectively. As can be seen in Fig. 21d, many parts of a crack have been disassembled and shown in different colors. While they belong to a crack and have been disconnected during image enhancement and noise reduction operations due to low image quality. Figure 22d shows a pseudo-color image of the crack segments after applying the retrieval method, which proves that presented method is greatly effective in alleviating this weakness.

Fig. 21

Identifying discrete crack segments BEFORE applying crack retrieval method. a RGB image; b binary image after image enhancement and denoising; c removing fine objects from the image; d pseudo-color or color-mapped image

Fig. 22

Identifying discrete crack segments AFTER applying crack retrieval method. a Extracting cracks skeleton of Fig. 21c (endpoints are shown as red); b cracks skeleton after applying the retrieval method; c retrieved cracks with real thickness; d pseudo-color or color-mapped image. (Color figure online)

5 Results and Discussion

The main goal in this paper was to design an autonomous crack recognition system to classify cracks from non-cracks and a type detection system to distinguish the type of cracks automatically. This was done by designing an intelligent agent-based fuzzy logic system based on an optimization of these fuzzy systems using PSO, all membership functions tuned and final shape of mfs determined. The outputs and results of this study are presented as follows.

5.1 Crack Detection

Figures 23, 24 and 25 represent respectively the mfs of Shape, Spalling, and Crack detection fuzzy inference systems, which are achieved from the optimization of these systems using PSO. The PSO convergence characteristic of fitness function in crack detection fuzzy system is shown in Fig. 26. Details of the coefficients of these functions are presented in Table 6.

Fig. 23

Rule viewer of SHAPE fuzzy inference system

Fig. 24

Rule viewer of SPALLING fuzzy inference system

Fig. 25

Rule viewer of CRACK DETECTION fuzzy inference system

Table 6 Optimized coefficients of mfs belong to hybrid crack detection fuzzy system

As explained in the description of the design of the fuzzy system in Sect. 2.1, 70% of the database in each class (here, two classes of cracks and non-cracks) are considered as the training set and the rest for testing the fuzzy system. As shown in Table 7, the number of data in class 1 (crack) is 291 and for class 2 (non-crack) is 846 images.

Table 7 Details of the optimization dataset of crack detection fuzzy system

Fig. 26

The PSO convergence characteristic of fitness function in crack detection fuzzy system

The testing set of these classes consists of 87 and 253 images, respectively. Figure 27 shows the confusion matrix and the results of applying the designed fuzzy system on these testing data sets. As can be seen, in the process of testing the crack recognition system, 86 out of 87 images of class 1 have been correctly detected as cracked (TP) and only one image has been found to be non-cracked inaccurately (FN). Also, 2 of 253 images of class 2 has been identified with cracks (FP) and the rest have been correctly identified as non-cracked (TN).

Fig. 27

Confusion matrix for testing dataset of crack detection fuzzy system

Based on these results, the accuracy and error of classification (cracks/non-cracks) in the designed fuzzy system are 98.24% and 1.76%, respectively, which derived from Eq. (19) and indicate satisfaction and high reliability of the presented system.

$$\begin{aligned} & {\text{Classification}}\;{\text{Rate}}/{\text{Accuracy}}\,(\% ) = \frac{{{\text{TP}} + {\text{TN}}}}{{{\text{TP}} + {\text{TN}} + {\text{FP}} + {\text{FN}}}} \\ \, \\ & {\text{Classification}}\;{\text{Error}}\,(\% ) = 1{-}{\text{Classification}}\;{\text{Rate}}\,(\% ) \\ \end{aligned}$$

(19)

5.2 Crack Type Detection System

The fuzzy mfs related to the crack classification (Mesh, Lengthy, Multiplicity, Deviation and type detection FIS’s) are illustrated in Fig. 28, 29, 30, 31 and 32, respectively. Table 8 also shows numerical values for the membership function coefficients obtained by applying the PSO on the designed FIS. The PSO convergence characteristic of fitness function in crack detection fuzzy system is shown in Fig. 33. The number of data in these classes is 192, 58 and 41 images, belonging to individual, pattern and random cracks (classes 1–3), respectively. The training data set contains 70% of this number and the remaining 30% used for testing. More information is provided in Table 9.

Fig. 28

Rule viewer of MESH fuzzy inference system

Fig. 29

Rule viewer of LENGTHY fuzzy inference system

Fig. 30

Rule viewer of MULTIPLICITY fuzzy inference system

Fig. 31

Rule viewer of DEVIATION fuzzy inference system

Fig. 32

Rule viewer of CRACK CLASSIFICATION fuzzy inference system

Table 8 Optimized coefficients of mfs belong to hybrid crack classification fuzzy system

Table 9 Details of the optimization dataset of crack classification fuzzy system

Fig. 33

The PSO convergence characteristic of fitness function in crack classification fuzzy system

The confusion matrix of the testing system on the dataset is illustrated in Fig. 34. Based on these results, the accuracy of identifying individual, pattern and random cracks is 98.25%, 95.45%, and 87.5%, respectively. The accuracy and error of the crack classification by the presented type detection system is 96.55% and 3.45%, respectively.

Fig. 34

Confusion matrix for testing dataset of crack classification fuzzy system

The 3D views of rule base of Shape, Spalling and Crack Detection FISs are shown in Fig. 35. These diagrams indicate the relation between the output and inputs of each fuzzy inference system. Figure 36 also displays the 3D viewer of rules designed for Crack Classification FIS for all three output classes include Individual, Pattern and Random cracks. These outcomes are from optimization of designed crack detection and classification fuzzy systems by PSO method.

Fig. 35

3D rule viewer plot of Shape, Spalling and Crack Detection fuzzy inference systems

Fig. 36

3D rule viewer plot of Crack Classification fuzzy inference system (for three crack types of Individual, Pattern and Random)

To more apperceive, the visual outputs of analyzing some cracked images (in all three types) by the presented method are displayed in Fig. 37.

Fig. 37

Image analysis results by the presented method (I1001 to I1003, P1001 to P1004 and R1001 to R1004 are individual, pattern and random crack samples, respectively). a Original RGB image; b binary image after image enhancement; c image stitching and denoising result; d pseudo-color image; e smoothened image of detected cracks. (Color figure online)

As explained in Sect. 3.2.1, the number of crack groups should be defined to achieve Multiplicity value in crack classification process. For this purpose, cracks belonging to a group are recognized and considered as a discrete collective unit at first. A new method is designed in this paper to merge all components of a crack group. Figure 38 shows the result of applying the Merging Algorithm on a random cracked image. As can be seen, the skeleton of each crack dilated in proportion with the skeleton length to specify its neighborhood domain and neighbor cracks. Each crack that exists as partial or full in the neighborhood domain is defined as a neighbor. The neighborhood matrix is a symmetric matrix included of 0 and 1, in which the neighbor cracks are assigned by 1 and the others by zero (Fig. 38g). Finally, the neighbor cracks that are merged together as a chain (see Fig. 38h), are considered as a unit crack group as shown in Fig. 38i.

Fig. 38

Merging neighbor cracks in image R1003. a Labeled cracks in pseudo-color image before applying Merging Algorithm; b–f showing neighboring area of each crack; g neighboring matrix of cracks; h biograph viewer of neighborhood matrix (each node illustrates a labeled crack in a that connects with its neighbor cracks); i pseudo-color image after applying Merging Algorithm. (Color figure online)

6 Conclusion and Future Work

Detection and classification of cracks as the most important distress in bridges received more attention during the past decades. Visual inspection is the most common traditional method, which is time consuming, costly, unsafe, and subjective, and might not be accurate enough. The main subject of this paper is to provide brief review a multi-agent fuzzy system (MAFs) based on image analysis for the detection and classification of, various types of cracking in concrete elements.

In this study, a new agent-based method been developed to detect and classify cracks in concrete bridges surfaces. The main contributions of this paper are developing A Multi-Agent Fuzzy system for crack detection and classification. For this purpose, the combination of fuzzy inference systems is developed and for making an autonomous agent, an intelligent method designed in the center of MAS to communicating information. The crack detection system and classification designed based on fuzzy MAS. The first agent made of five inputs, one output, 11 rules. The image as an input classified into crack and non-crack. The second agent, which is made of eight inputs, 3 output, 20 rules used for crack classification (individual, pattern and random). The input of this module is the image classified by type. The Particle Swarm Optimization (PSO) has been used to find the optimal values of membership functions. The optimized results of MAS are compared with other methods. After experimental testing of MAFs, the results show a high ability of MAFs for crack detection and classification. Analysis of the results showed that accuracy of detection, classification improved 4% and 5%, respectively. This method enhances the speed, accuracy and has higher precision, which indicates the satisfaction, and reliability of the MAFs. In addition, this system has the high computational power to detect and classify complex cracking patterns in bridge components.

As part of future work, the other kinds of defects appearing in concrete surfaces such as scaling, spalling and pop out needs to be evaluated. Therefore, by quantifying all types of distresses and the level of severity and extent, it will be possible to provide an overall distress index, which plays a vital role in the assessment of the structural health condition.