1 Introduction

Intelligentized robotic welding systems are one of the hot research branches in the welding field [1]. To realize the real automatic welding process, a great number of operations must be implemented such as weld quality monitoring, seam tracking, and decision making of welding process parameters [2, 3]. Many types of signals produced in the intelligentized robotic welding process (IRWP) can be used for implementing the above operations. Of these signals, visual information is the most widely used [4, 5], and structured light-based laser vision-sensing technologies are commonly used for robotic thick plate welding [6,7,8] to detect the weld seam profile. With the identified feature points of the extracted weld seam profile, the welding torch can be positioned for each pass and the real-time deviation of the welding torch can be eliminated with control algorithms. Therefore, weld seam profile extraction is a crucial step for subsequent operations. However, there are various adverse factors that influence image acquisition and result in the interference data surrounding the weld seam profile such as fumes, arc light, and splashes [9]. In addition, it is true that the larger depth of the groove of the workpiece the more inhomogeneous weld seam profile in terms of intensity, and the weld seam profile that crosses the groove is relatively weak. These are obstacles to integrally extracting the entire weld seam profile for each sampling time. Effective algorithms that can detect more details of the weld seam profile with higher ratios of effective extraction are continuously sought by researchers.

Extensive research concentrates on algorithms or schemes of weld seam profile extraction. These methods can be summarized in three types: traditional, visual attention–based, and target tracking methods. The related studies are reviewed as follows.

Traditional methods mean that some classic algorithms such as various filtering algorithms for denoising are used in the extraction process. In addition, searching algorithms for thresholding also belong to this category. For the denoising on the FPGA-based processing result, Pritschow et al. [10] presented a 2D high-pass filter to enhance edges and discontinuities in pixel intensity, and designed four special filter masks for detecting the weld seam profile. Moon et al. [11] eliminated noise caused by open arc and surroundings and used a pattern matching algorithm to detect the center position of the weld seam profile. To implement the multipass welding process based on visual sensing, a robust algorithm of weld seam profile extraction is proposed by Gu et al. [12], which consists of median filtering, thresholding, denoising, thinning, and curve fitting. Since the laser beam is strongly reflected from smooth metal surfaces, an intensity maximum searching algorithm is proposed to extract the weld seam profile in the region of interest by Nguyen and Lee [13]. A multiple peak detection algorithm similar to intensity-based searching methods for weld seam profile extraction is proposed by Chen et al. [14] to overcome arc interference in the high strength reflection area. An adaptive convolution procedure by Yin et al. [15] is presented, in which the weld seam profile center extraction method based on the geometry information and correlation is implemented, and followed with piecewise fitting to acquire seam profile centerlines. Traditional methods for weld seam profile extraction are relatively easy to be implemented and work effectively in most cases when arc light is kept out or the depth of the groove is relatively small.

Since the regular arc region contributes to positioning the welding torch in real time [16], it is useful except a barrier to weld seam profile extraction. Recently, visual attention models developed from visual attention mechanism are proposed for rapid scene analysis and image classification using saliency, which can also extract objects of interest from the complex background. Zhou et al. [17] proposed saliency detection to locate the weld pool region for weld seam extraction in the pipeline welding process. Gharsallah and Braiek [18] exploited an off-center saliency map as a feature to represent image pixels and realize weld defect inspection using radiography images. Li et al. [19] presented a visual saliency model based on local contrast to emphasize weld seam candidates for weld seam detection. Visual saliency evoked from this mechanism is also presented to emphasize the weld seam profile from the arc background [20, 21]. More complicated visual attention models investigated in a bottom-up (scene-dependent) manner are proposed to augment the saliency of the weld seam profile against the arc region [22, 23]. Because of the weld pool, back weld pool, and the seam in the same image, a visual attention–based method is also adopted to detect the small areas related to the weld pool by Yan et al. [24]. To eliminate heavy noise, Gong et al. [25] developed a multilayer hierarchy vision processing architecture integrated with bottom-up and top-down inference algorithms, in which the bottom-up process acts as a basic information collector while the top-down process performs acts as a missing data hunter. Visual attention–based methods use saliency to distinguish the weld seam profile from the arc background, and one defect of these methods is that interference with significant orientation characteristics (e.g., splashes) is simultaneously strengthened.

Currently, target tracking algorithms derived from visual attention are developed to track targets in images or videos. Classic target tracking algorithms use the correlation between adjacent images to determine the optical flow pattern. To extract the weld seam profile, a seam tracking system based on the morphological image processing method and a continuous convolution operator tracker (CCOT) algorithm are presented by [26]. These methods are also used to directly track an appointed feature point of the weld seam profile [27]. However, these algorithms cannot accurately position the entire weld seam profile because the profile possesses large spans in images. These methods are improper to track the feature point of the weld seam profile because the necessary tracking point changes with the position of the pass.

The weld seam image using active vision can be divided into two categories: the first category is that only the weld seam profile is in the image whereas the arc region is kept out, and the second one contains both. There are no algorithms that are fit for all applications to weld seam profile extraction, and the existing methods do not arrange a remedy for ineffective extraction. Note that empirical parameters account for ineffective extraction in most cases.

The study in this paper concentrates on two aims. The first one is striving to develop a general method to extract the weld seam profile for thick plate welding with the typical T-joints and butt joints, and the second one is attempting to build a fault detection and diagnosis (FDD) mechanism for weld seam profile extraction with a higher probability of successful extraction. To achieve the two targets, a method using top-down (task-dependent) visual attention model is first suggested, which fully uses the visual imaging characteristics of the weld seam profile and splashes to augment the saliency of the former. A dynamic saliency-based strategy is then presented to gradually discern the larger pieces of the weld seam profile in size through local competition at the initial stage. In the second stage, the exponentially weighted moving average (EWMA) control chart is investigated to overcome the disadvantage of the proposed method resulting from some empirical parameters used in this process and implement FDD.

2 Analysis of imaging characteristics of weld seam profiles and splashes

The spectrum generated in the metal active gas (MAG) arc welding process shows that the intensity of arc light is lower when its wavelength is from 630 to 660 nm. To maximize the intensity of the laser stripe (weld seam profile) as well as weaken arc light, the structured light-based laser whose central wavelength is about 660 nm is usually adopted. The weld seam profile is relatively weak when the entire arc region is not kept out (see Fig. 1a–c). On the contrary, using appropriate dimmer glass and optical filters, the weld seam profile is captured with higher intensity when the arc region is concealed (see Fig. 1d). The welding system and the visual sensing system refer to [23] and the imaging characteristic analysis of the weld seam profile consult [28].

Fig. 1
figure 1

Different raw images to show four imaging characteristics of laser stripes and splashes. ac The first three imaging characteristics of the weld seam profile with the arc region (the thickness of the workpiece is about 50 mm). d The fourth imaging characteristic with stronger weld seam profile in terms of intensity when the arc region is kept out (the thickness of the workpiece is 30 mm)

There are four characteristics of weld seam profiles and splashes that are used as target-driven factors to enhance the saliency of the former. The first characteristic is that the weld seam profile usually covers larger space than splashes do in the horizontal direction. The second one is that the thickness of the weld seam profile in the vertical direction is nearly a constant (about 5 pixels on average) whereas the thickness of the splash is not uniform. The third is that splashes usually cross the weld seam profile rather than are parallel to it with higher probabilities. In addition, the intensity of the weld seam profile in images with the arc region is weaker than that of the weld seam profile in images without the arc region because a dimmer glass must be used to acquire the more regular shape of the arc region (more regular shape can more accurately position the welding torch).

3 The proposed top-down visual attention model

Classic computational models of human visual search have embraced the idea of a saliency map to accomplish preattentive selection [29], which are mainly based on bottom-up visual attention and mostly implemented for scene analysis with a generated saliency map. For accurate weld seam profile extraction, these models are not competent to such task, and the test results using three classic models are given in this section. Meanwhile, since our previously presented models are mainly based on data-driven, they are sensitive to splashes. However, top-down visual attention models can strengthen the weld seam profile along with weakening interference as possible. In this paper, a top-down visual attention model is developed with these characteristics described in Section 2, as shown in Fig. 2.

Fig. 2
figure 2

Architecture of the proposed top-down visual attention model

3.1 Orientation feature detection

Compared with the intensity of the weld seam profile, its orientation characteristics are more salient than the arc region’s. Therefore, orientation feature detection is mainly considered in the proposed model. Gabor filters are used to produce orientation features (Offline tests determine the appropriate filtering angles). Note that two sets of filtering angles are used for different thickness of the workpiece, namely {5°, −5°} for less than 30 mm and {±5°, 80°} for over 30 mm. Orientation feature detection results with different filtering angles are linearly integrated into the final orientation feature map.

3.2 Denoising

Because of intense electromagnetic disturbance occurring during the welding process, the background of the captured image is full of different noise, which always survives after thresholding is carried out. In this study, it is removed as [12] first.

3.3 Saliency map

Since the weld seam profile is usually divided into several pieces after thresholding, the nearest neighbor clustering is used to gather the data points as clusters. There are three metric methods that use imaging characteristics of weld seam profiles and splashes to further strengthen the saliency of the weld seam profile. The data points belonging to the same cluster have the same saliency. The first metric is called length metric, which is described as

$$ \overset{\sim }{L_i}={L}_i\left({L}_i/\frac{1}{n}\sum \limits_{i=1}^n{L}_i\right) $$
(1)

where Li is the maximum Euclidean distance between the two points of the ith cluster, n is the number of the cluster, and \( \overset{\sim }{L_i} \) is length saliency. This metric effectively enlarges the saliency of the pieces of the weld seam profile with larger lengths whereas fully lowers the saliency of the interference data points. Width metric is the second metric method, which intentionally uses the thickness (this statistical parameter is referred to as Tn) of the weld seam profile to further strengthen the weld seam profile as well as weaken the interference data points:

$$ {\tilde{T}}_i=\Big\{{\displaystyle \begin{array}{l}{\left(|{T}_i-2|\right)}^3,\kern0.36em {T}_i<2\\ {}{T}_i,\kern0.36em 2\le {T}_i\le 10\\ {}{\left({T}_i/\max \left({T}_i\right)\right)}^{-3},\kern0.36em {T}_i>10\end{array}} $$
(2)

where Ti represents the average thickness of the ith cluster. Moreover, width uniformity metric is developed to measure the saliency of each cluster via variance \( \frac{1}{m_i}\sum \limits_{j=1}^{m_i}{\left({T}_{i,j}-\overline{T_i}\right)}^2 \):

$$ \overset{\sim }{V_i}={L}_i/\frac{1}{m_i}\sum \limits_{j=1}^{m_i}{\left({T}_{i,j}-\overline{T_i}\right)}^2 $$
(3)

where mi is the column number of the ith cluster, Ti, j is the thickness of the jth column of the ith cluster, and \( \overline{T_i} \) is the average thickness of all clusters. The three saliency maps are normalized and summed into the final global saliency map S like [30]:

$$ S=\frac{1}{3}\left(N\left(\overset{\sim }{L_i}\right)+N\left(\overset{\sim }{T_i}\right)+N\left(\overset{\sim }{V_i}\right)\right) $$
(4)

where Ν(⋅) is a normalization operator, with which \( \overset{\sim }{L_i} \), \( \overset{\sim }{T_i} \), and \( \overset{\sim }{V_i} \) are normalized to 0–255, respectively. To show the effectiveness of the proposed method in this paper, three classic bottom-up based models are tested as shown in Fig. 3.

Fig. 3
figure 3

Comparison of saliency maps between classic bottom-up based models and the proposed one. a Raw images. b Saliency maps produced with the proposed method in [31], c in [30], d in [32], and e in this paper

Figure 3 shows these classic models are not fit to accurately enhance the saliency of the long and narrow weld seam profile, whereas the top-down–based method proposed in this paper can intentionally highlight the weld seam profile against the interference background.

4 Strategy of weld seam profile extraction via local saliency competition

It is unrealistic to extract the weld seam profile at once because the orientation saliency of the weld seam profile is different in various regions. Piecewise extraction is a solution to this problem. Using the generated saliency map and combing the visual attention process by our eyes, the pieces of the weld seam profile is extracted gradually. The related strategy is presented through local saliency competition containing two steps: discerning the main parts of the weld seam profile and determining its details (see Fig. 4).

Fig. 4
figure 4

Strategy of extracting the weld seam profile based on local saliency

It is crucial to correctly discriminate the first piece of the weld seam profile. The rule of determining the first piece is selecting the first cluster (from left to right) that is strengthened with all three metric methods.

4.1 Candidate clusters

Since only one cluster is judged as part of the weld seam profile from candidate clusters in each discrimination loop, the rule of determining candidate clusters needs to be instituted. Candidate clusters are from the clusters whose data points are on the right of the last discerned piece in each loop. If the number of these clusters that satisfy the above requirement in each loop is Tempnum, the number of candidate clusters NCC is set as follows: NCC = 3 if Tempnum ≥ 3, NCC = 2 if Tempnum = 2, and NCC = 1 if Tempnum = 1.

4.2 Competition using local saliency

In each loop, saliency is still used to implement segment extraction of the weld seam profile. The saliency of each candidate cluster is dynamic, which is described as:

$$ {S}_{t,j}^{\hbox{'}}=\left[\frac{\underset{j}{\max}\left({D}_{t,j}\right)}{D_{t,j}}+\frac{L_{t,j}}{\underset{j}{\max}\left({L}_{t,j}\right)}\right]{S}_{t,j} $$
(5)

where j(j = 1, 2, ⋯, NCC) is the number of candidate clusters in the tth loop, Dt, j is the smallest Euclidean distance between the last discerned cluster and the jth candidate cluster, Lt, j is the maximum distance between two points of the jth candidate cluster, and St, j is the global saliency of the jth candidate cluster produced with Eq. (4). The candidate cluster with the maximum dynamic saliency \( {S}_{t,j}^{\hbox{'}} \) is picked in each loop.

4.3 End rule of discerning weld seam profile

The end rule of discerning the weld seam profile is Cp = Cl (see Fig. 4 and p is the subscript used in the loop). The rule does not work when larger splashes lie on the right of the entire laser stripe. The FDD method proposed in this paper accounts for this case. The saliency map with bigger V-groove in Fig. 3 is used to show the extraction process (Fig. 5).

Fig. 5
figure 5

Example of the weld seam profile extraction process. a The extraction process of the main parts of the weld seam profile. b Details of the weld seam profile. c The final extraction result

To show the effectiveness of the proposed method in this paper, our two published methods are applied as shown in Fig. 3a. The extraction results in Fig. 6 indicate that the proposed one here is closer to a general solution to different weld seam profile extraction than the previous methods and can extract more details of the weld seam profile.

Fig. 6
figure 6

Comparison of weld seam profile extraction using three methods. a The result using the method proposed in [22]. b The result using the method proposed in [23]. c The result using the method proposed in this paper

5 EWMA-based FDD for weld seam profile extraction process

Due to process parameters used in the proposed method, it is necessary to respond to their ineffectiveness during the welding process. The goal of FDD in this paper is to ensure higher probabilities of weld seam profile extraction.

The slopes of the effectively extracted weld seam profiles of different passes with T-joints and butt joints are statistically analyzed, and their three characteristics are revealed. Firstly, the slopes of the initial part of the weld seam profile are nearly a constant 0 (butt joints) or a negative constant (T-joints). Secondly, for the butt joint, the slopes change only once from positive to negative or from negative to positive with relatively big variations except in the case of the backing weld. In addition, the slopes of the final part of the weld seam profile are also nearly a constant 0 (butt joints) or a positive constant (T-joints). These statistical characteristics are shown in Fig. 7, in which the slopes are calculated using Eq. (6).

$$ {k}_{j-7}=\frac{\sum \limits_{i=1,3,5,7}\frac{y\left(j-i\right)-y\left(j+i\right)}{x\left(j+i\right)-x\left(j-i\right)}}{4}\left(j>=8\right) $$
(6)

where j represents the data point and y(⋅) and x(⋅) are two coordinates of data points in images, respectively. Each slope is determined with 8 adjacent data points. Note that the points used to calculate slopes are from linear weld seam profiles like the images in Fig. 7. Figure 7 shows that slopes of the weld seam profiles of different passes gradually change, and the changes of the monotonous intervals of slopes are generally small apart from the backing weld. The FDD method proposed in this paper is based on these characteristics, and the structure of the proposed FDD method is shown as Fig. 8.

Fig. 7
figure 7figure 7

Slope characteristics of the weld seam profiles: group I with butt joints and group II with T-joints

Fig. 8
figure 8

Structure of the proposed FDD method

5.1 Performance indexes and fault detection rules

In this paper, three performance indexes are defined to monitor whether the extracted weld seam profile is effective for T-joints and butt joints. The first performance index is the mean of the first 50 slopes using the data points of the first cluster \( \overset{\sim }{C_1} \) whose number of data points is over 50, which is referred to as AVSF. The second index is defined with the mean of the last 50 slopes using the data points of the last cluster \( \overset{\sim }{C_l} \) and referred to as ASLF. The third one is the biggest length of the monotonous intervals of slopes called BLMI, in which the first slope and the last one must be contrary sign and they both deviate from 0 to at least 0.3. Table 1 gives these definitions.

Table 1 Definitions of performance indexes

For butt joints, the first and the last identified clusters contain the horizontal parts of the weld seam profile. Therefore, the benchmarks of performance indexes I and II are both set to 0 to monitor ineffective extraction. Performance index III is used to monitor ineffective extraction with the interference data for the middle extraction process and set to 2.0. For T-joints, the benchmarks of performance indexes I, II, and III are 0.4, 0.7, and 2.0, respectively.

The general diagnosis rule for Tn is that it should be narrowed down when performance index I and/or II deviate(s) from the corresponding benchmark(s). The rule for NCC is that it should be increased if performance III deviates from its benchmark. The rules are set for both butt joints and T-joints. Tables 2 and 3 list the detailed rules, in which “↑” and “↓” indicate the changes of the performance index compared with its benchmark. The sign “―” indicates no change. Moreover, “∨” is the logic OR operation.

Table 2 Fault detection and diagnosis rules for butt joints
Table 3 Fault detection and diagnosis rules for T-joints

It must be noted that the fault occurring in the middle of the extraction process is directly detected through large changes of slopes rather than EWMA because the change in this case is so much.

5.2 Fault detection using EWMA

The EWMA method was originally presented by Roberts [33] for detecting small shifts described as Eq. (7). Thomson et al. [34] introduced EWMA control charts for the early detection of fouling of the heat recovery system. Zhao et al. [35] used the EWMA control chart to detect faults in a statistical way to improve the ratios of correctly detected points for centrifugal chillers. In this study, the EWMA control chart is used to detect the deviations of slopes of the designated part of the weld seam profile from the benchmarks.

$$ {\displaystyle \begin{array}{c}{Z}_i=\lambda \overline{X_i}+\left(1-\lambda \right){Z}_{i-1}\\ {}=\lambda \overline{X_i}+\left(1-\lambda \right)\overline{X_{i-1}}+\lambda {\left(1-\lambda \right)}^2\overline{X_{i-2}}+\cdots +\lambda {\left(1-\lambda \right)}^{i-1}\overline{X_1}+{\left(1-\lambda \right)}^i{\mu}_0\end{array}} $$
(7)

where \( \lambda \left(0<\lambda \le 1\right) \) is a constant weighting factor for determining the importance of the current group mean \( \overline{X_i} \) (the mean of ith sample group which has constant size n) and Zi is the ith EWMA value. Z0 = μ0 is the expected value (namely the corresponding benchmark). The control limits used in the EWMA control chart are calculated using Eqs. (8) and (9):

$$ \mathrm{UCL}={\mu}_0+ L\sigma \sqrt{\frac{\lambda }{n\left(2-\lambda \right)}} $$
(8)
$$ \mathrm{LCL}={\mu}_0- L\sigma \sqrt{\frac{\lambda }{n\left(2-\lambda \right)}} $$
(9)

where UCL and LCL are the upper and lower control limit respectively; L is the coefficient of the control limit; σ is the standard deviation of samples. n is set to 50. L and λ are influenced with average run lengths (ARL) when monitoring targets are in control and out of control. Statistical results suggest that λ should be small, which contributes to detecting tiny shifts with better ARL. L and λ depend on σ and the desired UCL/LCL.

6 Validation of the proposed FDD method

For butt joints, UCL is set to 0.23 for AVSF and ASLF using the characteristics of slopes of splashes. L is assigned to 75 when λ is 0.5 and σ is 0.04. To validate the proposed FDD method, 186 images are tested offline and the results show that the ratio of effective weld seam profile extraction increases from 92.4 to 98.6% (see example in Fig. 9).

Fig. 9
figure 9figure 9

Examples of the FDD process using the EWMA chart for butt joints (red circles mark examples of fault detection, similarly hereinafter). a Example of the FDD process for the initial stage of weld seam profile extraction. b Example of the FDD process for the final stage of weld seam profile extraction. c For the middle stage

For T-joints the test of the third pass is conducted, in which L is set to 71 and UCL is set to 0.8 when λ is 0.5 and σ is 0.025. In this case, faults usually occur in the middle and end stages of the weld seam profile extraction process. With the proposed FDD method, the ratio of successful extraction rises from 91.8 to 97.3%. Another welding test is done to show the influence of the FDD process on welds. Figure 10 illustrates two typical diagnosis processes and gives two welding results, which shows that the welding process with the proposed FDD in this paper is more stable than the process without this FDD process.

Fig. 10
figure 10figure 10

Examples of the FDD process using the EWMA chart for T-joints. a FDD for the initial stage of weld seam profile extraction. b FDD for the final stage of weld seam profile extraction. c Feature points of the weld seam profile using the method proposed by He et al. [23] (“point 3” is selected as the tracking point). d Welding result with better welds using the proposed FDD method. e Welding result without the proposed FDD method

7 Discussion

In this paper, we investigate a FDD method based on EWMA for improving the success probability of weld seam profile extraction during the traditional MAG welding process with thick steel plates. Our results show that the slopes of the weld seam profile seem to be an effective factor to determine whether the extracted weld seam profile is effective and the EWMA control chart is competent for implementing FDD by monitoring irregular changes of these slopes.

To our knowledge, this is the first FDD method for weld seam profile extraction with slopes in the robotic arc welding field with thick steel plates. This finding is significant for smoothing the welding process and improving welding quality, which offers useful references when laser vision sensing is used and empirical parameters are set in visual information extraction technologies.

To date, various visual sensing systems have been reported and many algorithms have been proposed for weld seam profile extraction. These algorithms [4,5,6] are used to cope with the case in which the thickness of the steel plate is less than 30 mm and the intensity of the laser stripe is high and uniform even in the groove region. It is relatively easy to extract the weld seam profile in this case. However, when the thickness increases to more than 50 mm, the intensity of the laser stripe becomes nonuniform, which brings about the issue described in this paper. Therefore, the segmental extraction process like this paper is an effective solution to weld seam profile extraction. Differing from [21,22,23], the top-down visual attention in this paper is implemented with the binary result of the final global saliency map whereas the previous visual attention methods are designed with the law image. The metric methods in this paper are also different from the previous ones.

Empirical parameters are usually used in various algorithms [12, 15, 21,22,23] for weld seam profile extraction. Different noise and unknown interference may hit the defects of these empirical parameters because the welding process is full of randomness. The FDD process to effectively adjust these parameters to adapt to unknown noise and interference is necessary. Currently, there is a little research that proposes methods to detect the faults because of empirical parameters for weld seam profile extraction during the welding process. We previously used the phenomenon of over proximity of the identified feature points of the extracted weld profile to trigger the FDD process [36]. Using the abnormal change of slopes in this study saves time cost in contrast to the previous one because the step of identifying the feature points of the extracted weld seam profile (using slopes) is not involved.

The method in this paper exposes its defects because of thresholds still used for UCL/LCL, λ, etc. These settings account for the rest percentage of effective weld seam profile extraction. Developing more effective methods and investigating more effective monitoring signals (e.g., the derivative of the slope) are our next research interest. Meanwhile, the corresponding strategy of adjusting the empirical parameters is studied in the future.

8 Conclusions

In conclusion, the proposed target-driven visual attention model combining the EWMA-based FDD method is used to extract weld seam profiles for the robotic MAG welding process with thick steel plates. Conclusions are as follows:

  1. 1.

    The abnormal changes of slopes of the extracted weld profile are a relatively effective monitoring object to determine whether the weld seam profile extraction process is effective.

  2. 2.

    With the proposed FDD method in this paper, the percentage of successful weld seam profile extraction is over 97% for the typical butt joints and T-joints.

  3. 3.

    Using this proposed method, a more stable welding process with better welds is acquired.

  4. 4.

    The proposed method in this work is nearly a general solution to weld seam profile extraction using the traditional welding methods to join thick steel plates with laser vision-sensing technologies. The method here shows a potential industrial application to the field of intelligentized robotic arc welding.