Accurate and robust sub-pixel refinement for fillet weld joint based on directional maximum projection

Abstract

Embedded seam tracking system that integrates sensing and computing hardware has obvious advantages due to its miniaturization and flexibility. A core problem of the embedded seam tracking system is to obtain accurate welding seam with low-resolution structured-light images. To remove this burden, an accurate and robust sub-pixel refinement method for fillet weld joint is proposed, and it mainly includes: initial direction detection of structured-light stripes by adopting mean shift algorithm, direction refinement based on weighted least squares method with constraint, and fillet weld joint sub-pixel refinement based on directional maximum projection. The main novelties of this work include: (1) the idea of refining fillet weld joint based on directional maximum projection is proposed for the first time; (2) the proposed method is accurate and robust in heavy noise images. The proposed method is robust, universal, and accurate, and as demonstrated by the following performances: (1) average biases of noise-free, rusty, highly reflective and arc-and-spatter images are 0.51, 0.55, 0.68, and 1.21 pixels, respectively; (2) the CR (effective rate) of pure, rusty, highly reflective, and arc-and-spatter images are all 100%; and (3) the offsets of the initial pixel-level fillet weld joint are effectively rectified.

1 Introduction

Welding is widely used in manufacturing owing to the advantages of economical materials and strong adaptability to various geometric shapes. Robotic welding has become a general trend because it can improve efficiency, quality, and protect workers from radiation. Be that as it may, the traditional welding robot based on “teach and playback” mode cannot adapt to welding deformation and dimensional deviation. The limitations can be solved by using welding seam–tracking technologies [1,2,3,4]. Seam tracking based on vision sensing [5], arc sensing [6], laser sensing [7], and electromagnetic sensing [8] has been researched intensively. Among them, vision sensing method dominates all sensing methods, accounting for approximately 64.7% [1].

Vision sensing method can be divided into structured-light vision sensing (active vision sensing) and passive vision sensing methods. The passive vision sensing method directly captures and detects the welding seam under natural light or welding arc. Because of the interference of arc, splash, and fume, the false detection of passive vision sensing method often happens. The structured-light vision sensing system projects a structured light onto the welding assembly and positions the welding seam by recognizing the structured-light feature points. Structured-light vision sensing method transforms the various weld forms into structured-light feature points (Fig. 1), so it is robust, universal and reliable [7, 9, 10].

Fig. 1

Schematic diagram of fillet weld joint detection under structured-light vision sensing system

At present, intelligent welding robot based on structured-light vision sensing method has become popular due to its reliability, robustness, and universality. A typical intelligent welding robot based on structured-light sensing includes the welding robot body, control system, structured-light vision system, and computing equipment. To improve the positioning accuracy of welding seam, it is usually necessary to use a large-resolution image acquisition system and high-performance computing equipment, resulting in increased costs.

In addition, the embedded seam tracking system that integrates sensing and computing hardware has obvious advantages due to its miniaturization and flexibility. But the arithmetic capability of embedded seam tracking system is generally poor. Therefore, many low-resolution image acquisition systems are adopted in embedded seam tracking system to reduce the burden on computing hardware. However, low-resolution images will reduce the positioning accuracy of welding seam. Therefore, realizing accurate and efficient weld positioning with low-resolution images is an urgent problem of the embedded seam tracking system.

Sub-pixel refinement can break through the limitation of physical resolution [11], so it has the potential to accurately positioning welding seam with low-resolution images, meeting the needs of the embedded seam tracking system. In addition, the hardware costs of the embedded seam tracking system can also be reduced through refining the welding seam. Generally, the moment-based methods [12,13,14,15], the interpolation-based methods [16,17,18], and the fitting-based methods [19, 20] are common choices for sub-pixel refinement. However, in our test, these methods cannot effectively refine the welding seam in the structured-light images, which often have significant errors.

There is only a small amount of works studying the sub-pixel refinement of welding seam. In the study of welding seam recognition based on passive vision sensing, Chen et al. presented a sub-pixel edge detection algorithm based on Zernike moments to locate welding seam [21]. The experiment results on S-shape and saddle-shape welding seam show that the presented method can be used for acquiring welding seam dimensional position in welding robot system. Shah et al. adopted skeleton extraction and edge model to generate sub-pixel type butt welding joint edge [22]. However, these sub-pixel refinement methods under passive vision are not applicable to refine welding seam under structured-light vision.

In the literature on welding seam recognition based on structured-light vision, there are a little works on welding seam sub-pixel refinement. Wang et al. adopted a gray centroid and random sampling consensus algorithm (RANSAC) to locate the initial welding seam in sub-pixel level [4]. Ma et al. also adopted the gray centroid and RANSAC algorithm to locate the start point guiding [23]. Experimental results show that these methods can achieve sub-pixel extraction, and the accuracy is better than that of the ECO method. Li et al. adopted the sequence gravity and recursive least squares method to extract welding seam in sub-pixel level [24]. The superiority of this algorithm is well demonstrated for seam tracking and recognition through extensive experiments. Lü et al. proposed an improved direction template and least squares method to position welding seam in sub-pixel level [25]. Firstly, the centerline of structured light is refined by the improved direction template. Secondly, the mathematical equation of the centerline is fitted by the least square method and the intersection is the welding seam. Zou et al. adopted the skeleton extraction and fitting method to extract the sub-pixel coordinates of the welding seam [26].

Table 1 shows the summary of the previous works. The moment-based methods are only suitable for refining the edges in the image, not for welding seam under structured-light image. The interpolation-based method requires a large interpolation density to achieve accurate sub-pixel refinement, resulting in a large amount of calculation. Therefore, this method does not match the embedded seam tracking system. The gray centroid method and skeleton method belong to the fitting-based method, which are accurate for noise-free structured-light images. But for images with heavy noise such as arcs, splashes, the structured-light stripe is difficult to be segmented from the background, so these methods will fail in this case. The directional template method is fast, but it is easy to fall into an endless loop in heavy noise images. In our test, the above mainstream methods cannot sub-pixel refine fillet weld joint well.

Table 1 Summary of previous works on sub-pixel refinement

To accurately and robustly refine the fillet weld joint in heavy noise structured-light images, an accurate and robust sub-pixel refinement method based on directional maximum projection is proposed, which is robust, universal and accurate. The proposed method is capable of refining the fillet weld joint by the following technical scheme: (1) calculating the initial directions of the structured-light stripes by adopting the mean shift algorithm; (2) direction refinement based on weighted least squares method with constraint; and (3) sub-pixel refinement of fillet weld joint based on directional maximum projection.

The contributions of this study include: (1) an idea composed of structured-light stripe initial direction calculation, structured-light stripe direction refinement and fillet weld joint refinement based on directional maximum projection is proposed to accurately and robustly refine the fillet weld joint, which can refine fillet weld joint in heavy noise structured-light images. (2) The sub-pixel refinement method based on directional maximum projection is proposed for the first time. In contrast to the existing approaches, the proposed method bypasses the technical framework of extracting the centerline of structured light, so it does not segment the structured-light stripe from the background. Therefore, the proposed method has obvious advantages in the working conditions of arc-and-spatter and highly reflective interference. This paper is organized as follows: The next section illustrates the sub-pixel refinement for fillet weld joint with the proposed method in detail. In Sect. 3, the tests and result analysis are presented. The conclusions are drawn in Sect. 4.

2 Methodology

The fillet weld joint is the intersection of the structured-light stripes in the structured-light images, so the positioning of the fillet weld joint is transformed into the extraction of the structured-light intersection. Generally, structured-light images contain multiple types of noise and interference such as arcs, splashes, reflections, and rust. Therefore, the sub-pixel refinement method must be robust to noise and interference. Figure 2 shows several typical structured-light images of the fillet weld joint. It can be seen that only the pixels on the structured-light stripes are useful for refining the fillet weld joint, while the rust, reflections, and arc spatter in the background are all interference factors.

Fig. 2

Four typical structured-light images of the fillet weld joint. (a) Pure. (b) Rusty. (c) Highly reflective. (d) Arc-and-spatter

To refine the fillet weld joint under heavy noise, an accurate and robust sub-pixel refinement method based on directional maximum projection is proposed. The pixels on the structured-light stripes are selected to refine the fillet weld joint, while the noise and interference in the background are filtered out. Therefore, the proposed method has strong anti-noise ability and robustness. The working process of the proposed method is as follows:

• 1. Intercept the (2n + 1) × (2n + 1) neighborhood image I with the initial pixel-level fillet weld joint P_O as the center (Fig. 3b). Empirically, a large neighborhood radius (n) can improve accuracy, yet the calculation will be cumbersome. n is 100 pixels in this paper.

• 2. Remove as much noise and interference as possible by using a linear enhancement filter kernel (Fig. 3c).

• 3. Calculate the initial directions of the structured-light stripes O₁ and O₂ by adopting the mean shift algorithm (Fig. 3d). The directions of the structured-light stripes is the basic constraint that selects pixels for refining fillet weld joint, and it is also the foundation of the sub-pixel refinement algorithm based on directional maximum projection. To obtain the initial directions of the structured-light stripes, the neighborhood image I is converted into a matrix of angle and intensity with the initial pixel-level fillet weld joint P_O as the center. Then, the mean shift algorithm is adopted to detect the initial direction O₁ and O₂ of the structured-light stripes.

• 4. Refine the directions of the structured-light stripes ${\varvec{O}}_{1}^{R}$ and ${\varvec{O}}_{2}^{R}$ by using the weighted least square method and with constraint (Fig. 3e). The directions of the structured-light stripes directly affect the accuracy of the refined fillet weld joint. Therefore, it is necessary to obtain precise directions of structured-light stripes. And the coarse coordinates of the fillet weld joint P_C can also be obtained accordingly.

• 5. Calculate the fine sub-pixel coordinates of the fillet weld joint P_F through the sub-pixel refinement method based on directional maximum projection (Fig. 3f). As we all know, the vector from P_F to the pixels on the structured-light stripes should be approximately parallel to its corresponding refined direction O^R. Based on this fact, we proposed the sub-pixel refinement method based on directional maximum projection and derive the mathematical formula.

Fig. 3

Working process of the proposed method. (a) Original image. (b) Neighborhood image. (c) Filtered image. (d) Initial directions. (e) Refined directions. (f) Refined fillet weld joint

To refine the fillet weld joint accurately and robustly in the heavy noise images, it is necessary to accurately segment the pixels on the structured-light stripes from the strong noise background. We used the directions of the structured-light stripes as the basic constraint to segment the pixels for refining fillet weld joint, so the precise directions of structured-light stripes are the foundation for the sub-pixel refinement of the fillet weld joint. Therefore, in Sects. 2.1 and 2.2, we will illustrate the method of calculating the directions of structured-light stripes in detail. The proposed sub-pixel refinement method based on directional maximum projection is expounded in Sect. 2.3.

2.1 Initial directions detection of structured-light stripes by adopting mean shift algorithm

The structured-light images usually contain noise and interference such as arcs, splashes, reflections, and rust, and the directions of the structured-light stripes are also uncertain, which brings challenges to the direction detection of the structured-light stripes. Inspired by the literature [27], the mean shift algorithm is adopted to detect the directions of the structured-light stripes. The mean shift algorithm is a parameter-free density estimation method, which is widely used in clustering, segmentation, and tracking. For n data points x_i (1 ≤ i ≤ n) in the d-dimensional space R_d, the basic form of the mean shift vector at the point x is:

$$M_{h} = {1 \mathord{\left/ {\vphantom {1 k}} \right. \kern-\nulldelimiterspace} k}\sum\limits_{{x_{i} \in {\varvec{S}}_{h} }} {\left( {x_{i} - x} \right)}$$

(1)

where S_h is a d-dimensional sphere with a radius of h, which satisfies the condition of S_h (x) = {y: (y − x)^T (y − x) ≤ h²}, and k is the number of points that fall in the area of S_h. Then make a new d-dimensional sphere with the end point of the mean shift vector as the center, and a new mean shift vector can be obtained. Repeating this step, the mean shift algorithm will converge to the point with the maximum density [28, 29].

In this paper, the mean shift algorithm in literature [28] is adopted to calculate the initial directions of the structured-light stripes, and the workflow is as follows:

• 1. Convert the neighborhood image I into a matrix of angle and intensity. For the pixel coordinate P (x, y), the vector form the initial pixel-level fillet weld joint P_O to P is P–P_O. First, calculating the angle θ (− π < θ ≤ π) between the vector P–P_O and the x axis; then, recording the gray level at pixel P (x, y) as S. I is converted into the matrix of θ and S.

• 2. According to the intensity threshold T_, the angle θ corresponding to the pixel P that satisfy the constraint of S_P > T are selected, forming the data set data (θ).

• 3. Cluster the data set data (θ) by using the mean Shift algorithm and calculate the center of each class. In this paper, the radius h = π/9, the minimum distance between clusters is h/2, and the threshold for end of cycle is T_stop = h/100.

• 4. According to the characteristics of the fillet weld joint images, the top two class centers are selected as the initial angle θ of the structured-light stripes. Then, the initial direction O (cosθ, sinθ) are calculated based on angle θ (Fig. 3d).

2.2 Direction refinement based on weighted least squares method

Noise and interference often cause deviations in the initial direction of the structured-light stripes detected by the mean shift algorithm. For example, in Fig. 3d, the noises in background are clustered into the pixel set of the structured-light stripes, causing the calculated initial directions of the structured-light stripes to deviate from their true directions (both the green and blue pixel sets contain some noise that is not on the structured-light stripes). According to the previous analysis, the accuracy of the structured-light stripe directions is directly affect the accuracy of the refined fillet weld joint. Therefore, it is necessary to rectify the directions of the structured-light stripes. The weighted least square method and with constraint is presented to refine the directions of the structured-light stripes, and the coarse sub-pixel coordinates of the fillet weld joint can also be obtained accordingly.

The core problem is which pixels should be selected for weighted least squares fitting. According to the previous analysis, only the pixels on the structured-light stripes are useful for fitting the directions of structured-light stripes. To avoid the noise and interference in the background from being selected, the weighted least squares fitting formula and with constraint is:

$$\begin{aligned} e &= \arg \min \sum\limits_{{P \in {\varvec{R}}_{i} }} {I\left( {x_{i} ,y_{i} } \right)} \left( {y_{i} - kx_{i} - b} \right)^{2} \hfill \\ s.t.{\text{ cos}}\left\langle {P - P_{{\text{O}}} ,{\varvec{O}}_{i} } \right\rangle &= {{\left[ {\left( {P - P_{{\text{O}}} } \right) \times {\varvec{O}}_{i} } \right]} \mathord{\left/ {\vphantom {{\left[ {\left( {P - P_{{\text{O}}} } \right) \times {\varvec{O}}_{i} } \right]} {\left( {\left| {P - P_{{\text{O}}} } \right|\left| {{\varvec{O}}_{i} } \right|} \right)}}} \right. \kern-\nulldelimiterspace} {\left( {\left| {P - P_{{\text{O}}} } \right|\left| {{\varvec{O}}_{i} } \right|} \right)}} \le 0.95 \hfill \\& \, \sqrt {\left| {P - P_{{\text{O}}} } \right|^{2} - \left| {\left( {P - P_{{\text{O}}} } \right) \times {\varvec{O}}_{i} } \right|^{2} } \le 5 \, I\left( {x_{i} ,y_{i} } \right) \ge T_{{\text{S}}} \hfill \end{aligned}$$

(2)

where e is optimal object; s.t. is constraint, R_i is the pixel set on the ith structured-light stripe; P = (x_i, y_i), I(x_i, y_i) is the gray level at pixel P; P_O is the initial pixel-level fillet weld joint; and O_i is the initial direction of the ith structured-light stripe (calculated in Sect. 2.1), and T_S is the intensity threshold.

For a structured-light stripe, the gray level of the center pixels is large, and the gray level of the edge pixels is small. To strengthen the contribution of the center pixels to the direction of the structured-light stripes, I(x_i, y_i) is used as the weighting coefficient. Three constraints are presented to select the pixels for refining the directions of the structured-light stripes, and their meanings are: (1) the angle between the vector P–P_O and the initial direction O_i is less than 18° (cos18° ≈ 0.95). (2) The distance from the pixel P(x_i, y_i) to O_i are less than 5 pixels; and (3) the gray level at pixel P(x_i, y_i) needs to meet I(x_i, y_i) > T_S.

Equation (2) is straightforward to be solved in closed form (The partial derivative of e is 0 at the minimum), yielding the solution:

$$\begin{aligned} \frac{\partial e}{{\partial k_{i} }} &= - 2\sum\limits_{{P \in {\varvec{R}}_{{\varvec{i}}} }} {I\left( {x_{i} ,y_{i} } \right)} \left( {y_{i} - kx_{i} - b_{i} } \right)x_{i} = 0 \hfill \\ \frac{\partial e}{{\partial b_{i} }} 7&= - 2\sum\limits_{{P \in {\varvec{R}}_{{\varvec{i}}} }} {I\left( {x_{i} ,y_{i} } \right)} \left( {y_{i} - kx_{i} - b_{i} } \right) = 0 \end{aligned}$$

(3)

Arrange Eq. (3) into a matrix form:

$$\left[ {\begin{array}{*{20}l} {\sum {I\left( {x_{i} ,y_{i} } \right)x_{i}^{2} } } \hfill & {\sum {I\left( {x_{i} ,y_{i} } \right)x_{i} } } \hfill \\ {\sum {I\left( {x_{i} ,y_{i} } \right)x_{i} } } \hfill & {\sum {I\left( {x_{i} ,y_{i} } \right)} } \hfill \\ \end{array} } \right]\left[ {\begin{array}{*{20}c} {k_{i} } \\ {b_{i} } \\ \end{array} } \right] = \left[ {\begin{array}{*{20}l} {\sum {I\left( {x_{i} ,y_{i} } \right)x_{i} y_{i} } } \hfill \\ {\sum {I\left( {x_{i} ,y_{i} } \right)y_{i} } } \hfill \\ \end{array} } \right]$$

(4)

k_i and b_i can be obtained by solving the matrix of Eq. (4). Based on $\theta_{i}^{{\text{R}}} = \arctan \left( {k_{i} } \right)$, the refined direction of the ith structured-light stripe ${\varvec{O}}_{i}^{{\text{R}}} \left( {\cos \theta_{i}^{{\text{R}}} ,\sin \theta_{i}^{{\text{R}}} } \right)$ can be calculated. The coarse sub-pixel coordinates of the fillet weld joint P_C can also be obtained accordingly, which is the intersection of the two fitted lines.

2.3 Sub-pixel refinement based on directional maximum projection

To obtain fine sub-pixel coordinates of the fillet weld joint, a sub-pixel refinement method based on directional maximum projection is proposed. As shown in Fig. 4, the sub-pixel fillet weld joint is located at the P_F, for neighboring pixels P_ij ∈ R_j ( j = 1, 2), the vector P_ij–P_F should be approximately parallel to its corresponding refined direction ${\varvec{O}}_{j}^{{\varvec{R}}}$. Assuming that S₁ is the cumulative sum of the projections of P_i1–P_F onto ${\varvec{O}}_{{1}}^{{\text{R}}}$, and S₂ is the cumulative sum of the projections of P_i2–P_F onto ${\varvec{O}}_{{2}}^{{\text{R}}}$, then S₁ + S₂ is the maximum when P_F is located on the actual sub-pixel fillet weld joint, leading to the optimization problem:

$$\begin{aligned} P_{{\text{F}}} &= \arg \min \sum\limits_{j = 1}^{2} {\sum\limits_{{P_{ij} \in {\varvec{R}}_{j} }} {\left( {\left| {{\varvec{P}}_{ij} \user2{ - P}_{{\text{F}}} } \right|^{2} - \left| {\left( {{\varvec{P}}_{ij} \user2{ - P}_{{\text{F}}} } \right) \times {\varvec{O}}_{j}^{{\text{R}}} } \right|^{2} } \right)} } \hfill \\ s.t.{\text{ cos}}\left\langle {{\varvec{P}}_{ij} \user2{ - P}_{{\text{C}}} ,{\varvec{O}}_{j}^{{\text{R}}} } \right\rangle &= {{\left[ {\left( {{\varvec{P}}_{ij} \user2{ - P}_{{\text{C}}} } \right) \times {\varvec{O}}_{j}^{{\text{R}}} } \right]} \mathord{\left/ {\vphantom {{\left[ {\left( {{\varvec{P}}_{ij} \user2{ - P}_{{\text{C}}} } \right) \times {\varvec{O}}_{j}^{{\text{R}}} } \right]} {\left( {\left| {{\varvec{P}}_{ij} \user2{ - P}_{{\text{C}}} } \right|\left| {{\varvec{O}}_{j}^{{\text{R}}} } \right|} \right)}}} \right. \kern-\nulldelimiterspace} {\left( {\left| {{\varvec{P}}_{ij} \user2{ - P}_{{\text{C}}} } \right|\left| {{\varvec{O}}_{j}^{{\text{R}}} } \right|} \right)}} \le 0.95 \hfill \\& \, \sqrt {\left| {{\varvec{P}}_{ij} \user2{ - P}_{{\text{C}}} } \right|^{2} - \left| {\left( {{\varvec{P}}_{ij} \user2{ - P}_{{\text{C}}} } \right) \times {\varvec{O}}_{j}^{{\text{R}}} } \right|^{2} } \le 5 \hfill \, I\left( {P_{ij} } \right) \ge T_{{\text{S}}} \hfill \\ \end{aligned}$$

(5)

where P_C is the coarse coordinates of the fillet weld joint (calculated in Sect. 2.2), j represents the jth structured-light stripe (j = 1, 2), R_j is the pixel set on the jth structured-light stripe, and i is the number of pixels on the jth structured-light stripe. This problem is straightforward to solve in closed form, yielding the solution (Appendix presents the detailed mathematical reasoning of the Eq. (6)):

$$P_{F} = \sum\limits_{j = 1}^{2} {\sum\limits_{{P_{ij} \in \varvec{R}_{j} }} {\left( {1 - O_{j}^{{\text{R}}} O_{j}^{\text{RT}} } \right)} }^{ - 1} \sum\limits_{j = 1}^{2} {\sum\limits_{{P_{ij} \in \varvec{R}_{j} }} {\left( {1 - O_{j}^{{\text{R}}} O_{j}^{{{\text{RT}}}} } \right)} } P_{ij}$$

(6)

Fig. 4

Schematic diagram of the proposed sub-pixel refinement method based on directional maximum projection

To verify the correctness of the proposed sub-pixel refinement method based on directional maximum projection, we tested it by using two generated images with known sub-pixel coordinates of the fillet weld joint (the coordinates of the fillet weld joint are both [101, 101]). The coordinates refined by the proposed method are both [101, 101], so the proposed sub-pixel refinement method based on directional maximum projection is correct.

3 Test and analysis

In this section, experiments are conducted to verify the proposed method. The experimental equipment mainly included a welding robot, a red laser structured-light vision system with an image size of 615 × 735, and a computer with a 2.6 GHz Intel Core i5–11400F CPU. The image processing software is Matlab2021a. The four core indicators of the fillet weld joint sub-pixel refinement algorithm are the accuracy, efficiency, universality and robustness. In Sect. 3.1, we compared the accuracy and efficiency of methods A, B, and the proposed method, and the universality, robustness, and offset correction capability are tested in Sect. 3.2.

Method A [23] adopted the gray centroid method and random sampling consensus algorithm (RANSAC) to locate the start point guiding. Firstly, the gray centroid method is used to refine the structured-light centerline in sub-pixel level; secondly, the RANSAC algorithm is adopted to fit the centerline. The intersection of the fitted centerline is the welding seam. Method B [25] presented an improved direction template and least squares method to position welding seam. Firstly, the image is segmented by using the Otsu’s method. Secondly, the centerline of structured light is refined by the improved direction template method. Finally, the mathematical equation of the centerline is fitted by the least square method, and the intersection is the welding seam.

3.1 Comparison of accuracy and computing time

Accuracy and computing time are the critical performance indices of the sub-pixel refinement method. Since the sub-pixel fillet weld joint in the structured-light images cannot be directly measured, we generated six structured-light images with known fillet weld joint sub-pixel coordinates. The size of the generated original images is 5000 × 5000 pixels, and the coordinates of the fillet weld joint in the original image are (2501, 2501) pixel. Then, the size of the original images is compressed from 5000 × 5000 to 500 × 500 pixels, so the theoretical sub-pixel coordinates of the fillet weld joint in the compressed image are (250.55, 250.55) (because the sub-pixel range of the compressed image is [0.5, 500.5]).

To make the generated images as similar to the actual images as possible, we added rust, reflective noise, and arc-and-spatter noise to the generated images respectively. Through above process, 24 structured-light images are generated for test (The number of noise-free, rusty, highly reflective and arc-and-spatter images are all 6). Figure 5 shows a group of compressed structured-light images under four working conditions.

Fig. 5

Compressed structured-light images. (a) Noise-free. (b) Rusty. (c) Highly reflective. (d) Arc-and-spatter

Table 2 shows the accuracy and computing time of methods A, B and the proposed method. After refining the fillet weld joint in the compressed images, the sub-pixel coordinates are converted to the coordinates in the original image of 5000 × 5000 pixels. Then the positioning bias is calculated in the original image (the bias is reduced by 10 times if it is calculated in the compressed image). The neighborhood radius used to refine fillet weld joint is 100 pixels of methods A, B, and the proposed method, and the image preprocessing used by methods A, B, and the proposed method are same.

Table 2 Accuracy and computing time of methods A, B and the proposed method

Methods A, B, and the proposed method are all accurate for noise-free structured-light images, which all can meet the requirements of sub-pixel refinement. Method B spends less calculating time, but method A and the proposed method can also meet the needs of real-time seam tracking. For noisy images such as rust, high reflection, and arc spatter, the proposed method has the best accuracy and robustness because it selects the pixels based on the directions of the structured-light stripes and the constraints. Method B is proposed for offline welding of V-groove welding seam (the structured-light images are pure), so it does not consider the interference of noise on welding seam detection. The ridge line tracking of method B often falls into an infinite loop in noisy images, so the structured-light stripes cannot be extracted stably. Method A is more robust than method B because it uses the RANSAC algorithm, but it still cannot stably refine the centerline of the structured-light stripes for heavy noisy images.

3.2 Tests of universality, robustness and offset correction capability

Universality and robustness are also the important performance parameters because the sub-pixel refining method needs to be useful to a variety of working conditions. Therefore, in Sect. 3.2.1, we compared the universality and robustness of methods A, B, and the proposed method, and the offset correction capability of the proposed method is tested in Sect. 3.2.2.

3.2.1 Universality and robustness test

To meet the requirements of real-time seam tracking, sub-pixel refining method not only needs to work on pure images, but also needs to be effective for noisy images such as rusty, highly reflective, and arc-and-spatter images. The universality and robustness of sub-pixel method is the capability to be effective in a variety of working conditions. The universality and robustness of methods A, B, and the proposed method are compared by using 80 structured-light images under four working conditions (20 pure, 20 rusty, 20 highly reflective, and 20 arc-and-spatter structured-light images).

Since the precise sub-pixel coordinates of fillet weld joint in structured-light image cannot be measured, correctness of the refined fillet weld joint is judged by comparing the deviation between the refined sub-pixel and the initial pixel-level coordinates. The initial pixel-level fillet weld joint is positioned by adopting the method in literature [30]. If the deviations of the refined and the initial pixel-level fillet weld joint are less than 4 pixels, the sub-pixel method is considered effective for the current image; otherwise, the sub-pixel method is considered invalid. Table 3 shows the CR (effective rate) of methods A, B, and the proposed method.

Table 3 CR (effective rate) of methods A, B, and the proposed method

Table 3 reveals that the proposed method is useful for 4 working conditions, and its CR under the four working conditions are all 100%, indicating that the proposed method has strong universality and robustness. Method A also has strong universality and robustness, and its CR of pure images is 100%. Although the arc-and-spatter images contain a lot of noises, the noises are randomly distributed. The RANSAC algorithm adopted by method A can filter them out, so its CR is high for the arc-and-spatter images. The reflection patterns of the highly reflective images are extremely misleading. Methods A and B often misidentify the reflection patterns as structured-light stripes, so the CR of the methods A and B for highly reflective images are the lowest among the four working conditions. Since the ridge line tracking often falls into an infinite loop, the CR of method B is low, so the method B is not suitable for our working conditions. But in the images that do not fall into the infinite loop, the refined fillet weld joint is accurate. Therefore, after proper improvement, the CR of method B will be greatly improved.

3.2.2 Offset correction test

In noisy structured-light images such as rusty, highly reflective, and arc-and-spatter images, there may be pixel-level offsets in the initial pixel-level fillet weld joint, which brings challenges to sub-pixel refinement. Theoretically, the fillet weld joint is in a certain position, so the refined fillet weld joint is the same no matter where the initial pixel-level fillet weld joint is. In practice, the initial pixel-level offsets will change the neighborhood pixels for refining fillet weld joint, resulting in different coordinates of the refined fillet weld joint. To obtain accurate sub-pixel fillet weld joint, the pixel-level offsets need to be rectified by the sub-pixel method. The offset correction capability of the sub-pixel method also represents its robustness.

The variation of the refined fillet weld joint reflects its offset correction capability when the initial pixel-level fillet weld joint is changed. Small variation means that the sub-pixel method has a strong offset correction capability. The initial pixel-level fillet weld joint is not used by methods A and B when refining fillet weld joint, so the offset correction capability of methods A and B does not need to be tested. We continued to use the 80 structured-light images to test the offset correction capability of the proposed method. Figure 6 presents the refined fillet weld joint by the proposed method with a structured-light image as an example when the initial pixel-level offsets are 0, 1, 2, 3, and 4 pixels, respectively (The red and green points are the initial pixel-level and refined sub-pixel fillet weld joints, respectively). It can be seen that the refined fillet weld joints are always near the same position, indicating that the proposed method has a strong offset correction capability.

Fig. 6

The refined fillet weld joint by the proposed method. (a) No offset. (b) Offset is 1 pixel. (c) Offset is 2 pixels. (d) Offset is 3 pixels. (e) Offset is 4 pixels

Figure 7 shows the test results about the 80 structured-light images. It reveals that the maximum and average variation of the refined fillet weld joint are 2.83 and 0.73 pixels in the 4 working conditions when the initial pixel-level offsets is within 4 pixels. The variation of the refined fillet weld joint increases with the increase of the initial pixel-level offset, because the effective neighborhood pixels for refining fillet weld joint reduces as the initial pixel-level offset increases (refer to Eq. (5)). The variation of the pure structured-light images is the smallest among the 4 working conditions, but the variations of the heavy noisy structured-light images are large. This is because the pixel set for refining the fillet weld joint in heavy noisy images contains more noises than pure images when the initial pixel-level fillet weld joint is deviated. The highly reflective images have the largest variation, because the reflection patterns cannot be filtered out by the linear enhancement filter kernel (Sect. 3) due to its similar features to the structured-light stripes.

Fig. 7

Offset correction capability of the proposed method. (a) Mean of variation. (b) Maximum of variation

On the whole, the advantages of this study are: (1) the proposed method is more accurate compared with other methods such as methods A and B and (2) the proposed method is robust and universal, so it can be applied to heavy noise working conditions. But it also has a disadvantage: The calculating time of the proposed method is longer than methods A and B, which limits its application in real-time seam tracking with high sampling frequency.

4 Conclusion

This paper presents an accurate and robust sub-pixel refinement method for fillet weld joint. The proposed method is designed for structured-light vision system and mainly consists of three steps: initial directions detection of structured-light stripes, direction refinement of structured-light stripes, and fillet weld joint refinement based on directional maximum projection. The main technical innovations of this work include: (1) a novel sub-pixel method is proposed for refining fillet weld joint, which is accurate, robust, and universal; (2) the idea of refining fillet weld joint based on directional maximum projection is proposed for the first time; and (3) the mean shift and weighted least squares method with constraint can obtain accurate directions of structured-light stripes.

Test results have verified that the proposed method could accurately and robustly refine the fillet weld joint, as demonstrated by the following main performance indices: (1) average biases of noise-free, rusty, highly reflective, and arc-and-spatter images are 0.51, 0.55, 0.68, and 1.21 pixels, respectively; (2) the CR (effective rate) of pure, rusty, highly reflective, and arc-and-spatter images are all 100%; and (3) the offsets of the initial pixel-level fillet weld joint are effectively rectified.

Although the proposed method could accurately and robustly refine the fillet weld joint, there are some limitations of this study: (1) the initial pixel-level coordinates of fillet weld joint must be obtained before using this method, so this method needs to be paired with a pixel-level fillet weld joint detection method (2) this method is designed for fillet weld joint and it may be effective for groove welding seam after improvement. But it is invalid for butt welding seam because the features of butt welding seam are different from the fillet weld joint under structured-light vision.

Appendix

As shown in Fig. 4, setting O_xi = cosθ_i and O_yi = sinθ_i, then the refined direction of structured-light stripe 1 is O₁^R = (O_x1, O_y1) and the refined direction of structured-light stripe 2 is O₂^R = (O_x2, O_y2). Assuming that the P_i1(x_i1, y_i1) is the selected pixel set on the structured-light stripes 1 (satisfy P_i1 ∈ R₁) and P_i2(x_i2, y_i2) is the selected pixel set on the structured-light stripes 2 (satisfy P_i2 ∈ R₂).

Set the coordinates of refined fillet weld joint P_F is (x_F, y_F), then Eq. (6) can be transformed into:

$$\begin{aligned} f &= \arg \min \sum\limits_{{P_{i1} \in {\varvec{R}}_{1} }} \Big[ \left( {x_{i1} - x_{{\text{F}}} } \right)^{2} \left( {1 - O_{x1}^{2} } \right) + \left( {y_{i1} - y_{{\text{F}}} } \right)^{2} \\&\left( {1 - O_{y1}^{2} } \right) - 2\left( {x_{i1} - x_{{\text{F}}} } \right)\left( {y_{i1} - y_{{\text{F}}} } \right)O_{x1} O_{y1} \Big] \\&+ \sum\limits_{{P_{i2} \in {\varvec{R}}_{2} }} \Big[ \left( {x_{i2} - x_{{\text{F}}} } \right)^{2} \left( {1 - O_{x2}^{2} } \right) + \left( {y_{i2} - y_{{\text{F}}} } \right)^{2} \\&\left( {1 - O_{y2}^{2} } \right) - 2\left( {x_{i2} - x_{{\text{F}}} } \right)\left( {y_{i2} - y_{{\text{F}}} } \right)O_{x2} O_{y2} \Big] \end{aligned}$$

(7)

According to extremum conditions that the partial derivative of f is 0 at the minimum, yielding the solution:

$$\begin{aligned} \frac{\partial f}{{\partial x_{{\text{F}}} }} &= - 2\left( {1 - O_{x1}^{2} } \right)\sum\limits_{{P_{i1} \in {\varvec{R}}_{1} }} {\left( {x_{i1} - x_{{\text{F}}} } \right)} + 2O_{x1} O_{y1} \sum\limits_{{P_{i1} \in {\varvec{R}}_{1} }} {\left( {y_{i1} - y_{{\text{F}}} } \right)} \\&- 2\left( {1 - O_{x2}^{2} } \right)\sum\limits_{{P_{i2} \in {\varvec{R}}_{2} }} {\left( {x_{i2} - x_{{\text{F}}} } \right) + 2O_{x2} O_{y2} \sum\limits_{{P_{i2} \in {\varvec{R}}_{2} }} {\left( {y_{i2} - y_{{\text{F}}} } \right)} } = 0 \\ \frac{\partial f}{{\partial y_{{\text{F}}} }} &= - 2\left( {1 - O_{y1}^{2} } \right)\sum\limits_{{P_{{_{i1} }} \in {\varvec{R}}_{1} }} {\left( {y_{i1} - y_{{\text{F}}} } \right)} + 2O_{x1} O_{y1} \sum\limits_{{P_{i1} \in {\varvec{R}}_{1} }} {\left( {x_{i1} - x_{{\text{F}}} } \right)} \\&- 2\left( {1 - O_{y2}^{2} } \right)\sum\limits_{{P_{i2} \in {\varvec{R}}_{2} }} {\left( {y_{i2} - y_{{\text{F}}} } \right) + 2O_{x2} O_{y2} \sum\limits_{{P_{i2} \in {\varvec{R}}_{2} }} {\left( {x_{i2} - x_{{\text{F}}} } \right)} } = 0 \hfill \\ \end{aligned}$$

(8)

Simplify and derive Eq. (8) to get the following equation:

$$\begin{aligned} &\left[ {n_{1} \left( {1 - O_{x1}^{2} } \right) + n_{2} \left( {1 - O_{x2}^{2} } \right)} \right]x_{{\text{F}}} - \left( {n_{1} O_{x1} O_{y1} + n_{2} O_{x2} O_{y2} } \right)y_{{\text{F}}} = \left( {1 - O_{x1}^{2} } \right)\\&\sum\limits_{{P_{i1} \in {\varvec{R}}_{1} }} {x_{i1} } - O_{x1} O_{y1} \sum\limits_{{P_{i1} \in {\varvec{R}}_{1} }} {y_{i1} } + \left( {1 - O_{x2}^{2} } \right)\sum\limits_{{P_{i2} \in {\varvec{R}}_{2} }} {x_{i2} } - O_{x2} O_{y2} \sum\limits_{{P_{i2} \in {\varvec{R}}_{2} }} {y_{i2} } \hfill \\& \left[ {n_{1} \left( {1 - O_{y1}^{2} } \right) + n_{2} \left( {1 - O_{y2}^{2} } \right)} \right]y_{{\text{F}}} - \left( {n_{1} O_{x1} O_{y1} + n_{2} O_{x2} O_{y2} } \right)x_{{\text{F}}} = \left( {1 - O_{y1}^{2} } \right)\\&\sum\limits_{{P_{i1} \in {\varvec{R}}_{1} }} {y_{i1} } - O_{x1} O_{y1} \sum\limits_{{P_{i1} \in {\varvec{R}}_{1} }} {x_{i1} } + \left( {1 - O_{y2}^{2} } \right)\sum\limits_{{P_{i2} \in {\varvec{R}}_{2} }} {y_{i2} } - O_{x2} O_{y2} \sum\limits_{{P_{i2} \in {\varvec{R}}_{2} }} {x_{i2} } \hfill \\ \end{aligned}$$

(9)

where n₁ is number of pixels in the pixel set R₁, and n₂ is number of pixels in the pixel set R₂. Equation (9) is written in matrix form as:

$$\begin{aligned}&\left[ {\begin{array}{*{20}l} {\sum\limits_{j = 1}^{2} {n_{j} \left( {1 - O_{xj}^{2} } \right)} } \hfill & { - \sum\limits_{j = 1}^{2} {n_{j} O_{xj} O_{yj} } } \hfill \\ { - \sum\limits_{j = 1}^{2} {n_{j} O_{xj} O_{yj} } } \hfill & {\sum\limits_{j = 1}^{2} {n_{j} \left( {1 - O_{yj}^{2} } \right)} } \hfill \\ \end{array} } \right]\left[ {\begin{array}{*{20}c} {x_{{\text{F}}} } \\ {} \\ {y_{{\text{F}}} } \\ \end{array} } \right] \\&= \left[ {\begin{array}{*{20}l} {\left( {1 - O_{x1}^{2} } \right)\sum\limits_{{P_{i1} \in {\varvec{R}}_{1} }} {x_{i1} } - O_{x1} O_{y1} \sum\limits_{{P_{i1} \in {\varvec{R}}_{1} }} {y_{i1} } + \left( {1 - O_{x2}^{2} } \right)\sum\limits_{{P_{i2} \in {\varvec{R}}_{2} }} {x_{i2} } - O_{x2} O_{y2} \sum\limits_{{P_{i2} \in {\varvec{R}}_{2} }} {y_{i2} } } \hfill \\ { - O_{x1} O_{y1} \sum\limits_{{P_{i1} \in {\varvec{R}}_{1} }} {x_{i1} + \left( {1 - O_{y1}^{2} } \right)\sum\limits_{{P_{i1} \in {\varvec{R}}_{1} }} {y_{i1} } } - O_{x2} O_{y2} \sum\limits_{{P_{i2} \in {\varvec{R}}_{2} }} {x_{i2} } + \left( {1 - O_{y2}^{2} } \right)\sum\limits_{{P_{i2} \in {\varvec{R}}_{2} }} {y_{i2} } } \hfill \\ \end{array} } \right]\end{aligned}$$

(10)

Simplify Eq. (10) to get the following matrix form as:

$$\begin{aligned}\sum\limits_{j = 1}^{2} {n_{j} } &\left( {\left[ {\begin{array}{*{20}c} 1 & 0 \\ 0 & 1 \\ \end{array} } \right] - \left[ {\begin{array}{*{20}c} {O_{xj} } \\ {O_{yj} } \\ \end{array} } \right]\left[ {\begin{array}{*{20}c} {O_{xj} } & {O_{yj} } \\ \end{array} } \right]} \right)\left[ {\begin{array}{*{20}c} {x_{F} } \\ {y_{F} } \\ \end{array} } \right] \\&= \sum\limits_{j = 1}^{2} {\left( {\left[ {\begin{array}{*{20}c} 1 & 0 \\ 0 & 1 \\ \end{array} } \right] - \left[ {\begin{array}{*{20}c} {O_{xj} } \\ {O_{yj} } \\ \end{array} } \right]\left[ {\begin{array}{*{20}c} {O_{xj} } & {O_{yj} } \\ \end{array} } \right]} \right)} \left[ {\begin{array}{*{20}c} {\sum\limits_{{P_{ij} \in {\varvec{R}}_{j} }} {x_{ij} } } \\ {\sum\limits_{{P_{ij} \in {\varvec{R}}_{j} }} {y_{ij} } } \\ \end{array} } \right]\end{aligned}$$

(11)

Substitute P_ij and O_j^R for (x_ij, y_ij) and (O_xj, O_yj), Eq. (11) can be transformed into:

$$\sum\limits_{j = 1}^{2} {n_{j} } \left( {1 - {\varvec{O}}_{j}^{{\text{R}}} {\varvec{O}}_{j}^{{{\text{RT}}}} } \right)\left[ {\begin{array}{*{20}c} {x_{{\text{F}}} } \\ {y_{{\text{F}}} } \\ \end{array} } \right] = \sum\limits_{j = 1}^{2} {\sum\limits_{{P_{ij} \in {\varvec{R}}_{j} }} {\left( {1 - {\varvec{O}}_{j}^{{\text{R}}} {\varvec{O}}_{j}^{{{\text{RT}}}} } \right)P_{ij} } }$$

(12)

According to the rules of matrix operations, the calculation formula of (x_F, y_F) is:

$$\left[ {\begin{array}{*{20}c} {x_{{\text{F}}} } \\ {y_{{\text{F}}} } \\ \end{array} } \right] = \sum\limits_{j = 1}^{2} {n_{j} } \left( {1 - \varvec{O}_{j}^{{\text{R}}} \varvec{O}_{j}^{{{\text{RT}}}} } \right)^{ - 1} \sum\limits_{j = 1}^{2} {\sum\limits_{{P_{ij} \in \varvec{R}_{j} }} {\left( {1 - \varvec{O}_{j}^{{\text{R}}} \varvec{O}_{j}^{{{\text{RT}}}} } \right)P_{ij} } }$$

(13)

Substitute P_F for (x_F, y_F), Eq. (13) can be transformed into:

$$P_{{\text{F}}} = \sum\limits_{j = 1}^{2} {\sum\limits_{{P_{ij} \in {\varvec{R}}_{j} }} {\left( {1 - {\varvec{O}}_{j}^{{\text{R}}} {\varvec{O}}_{j}^{{{\text{RT}}}} } \right)^{ - 1} } } \sum\limits_{j = 1}^{2} {\sum\limits_{{P_{ij} \in {\varvec{R}}_{j} }} {\left( {1 - {\varvec{O}}_{j}^{{\text{R}}} {\varvec{O}}_{j}^{{{\text{RT}}}} } \right)P_{ij} } }$$

(14)

Equation (14) can be expressed easily through programming.