1 Introduction

Weld seam recognition based on laser-structured light is considered one of the popular vision sensors in the applications of welding. It is used widely to detect different types of welding joints, such as narrow butt joints [1,2,3], tee-joints [4, 5], lap joints [6], and V-joints [7,8,9,10]. It is used widely to detect different types of welding joints, such as narrow butt joints [1,2,3], tee-joints [4, 5], lap joints [6], and V-joints [7,8,9,10].

The importance of obtaining the weld seam groove’s information comes in assisting to direct the welding torch to perform optimal welding for compensating the depth and width of the weld seam groove during the welding process [12]. Therefore, the detection algorithm should be capable of acquiring information about the position and geometrical features of the weld seam groove [13]. For extracting the image features of the weld seam groove, there are three sub-processes that need to be applied. Firstly is image preprocessing, which is commonly used for removing the noise and identifying the region of interest. Secondly is the extraction of the laser line profile. Lastly is the corner points extraction of the weld seam groove [14].

In the phase of image preprocessing phase, the reflection of the surface, arc, and other welding noise needs to be removed. Image denoising and enhancement will therefore be used in this phase, which is a crucial step for identifying the weld seam in a welding setting. There are many methods that were previously used for image enhancement and denoising; the median filter [15] is one of these methods that have the ability to preserve the details of the edge of the weld image and remove the pepper and salt noise. The frequency noise in the image is reduced using the Gaussian filter [11]. Some researchers employ the morphological filter to enhance the quality of the images [16].

The second phase is the recognition of the laser line profile of the weld seam groove. The form and surface of the workpiece of the weld seam groove have a major role in the extraction of the laser stripe. Because the laser stripe in the image will probably have a variable width, the center line of the laser stripe must be extracted after the laser line. Radon transformation [17] and Hough transformation [13] [18,19,20] are one of the methods for extracting the central line of the laser stripe, where the laser centerline can be fitted by them. Li et al. [15] extracted the laser central line by adopting inside stalk transformation. Since the laser stripe region has the maximum pixel intensity in the image, one way for extracting the laser center line is to look for the peak pixel in each column or row of the image [21, 22]. Huang and Kovacevic [11] extracted the laser line using the approach of second central difference of each pixel’s row index on the laser stripe. Some subpixel extraction methods, such the center of gravity approach [24] and the Gaussian approximation method [23], are also employed to extract the location of the laser centerline in order to improve detection accuracy even more. Additionally, the laser center line was extracted using the edge detection technique [25]. Others have used Laplacian of Gaussian filter (LoG) [26] for searching each image’s column or row and calculate the result. The LoG filter takes into account the laser stripe’s width information so that some welding spatter noise interference can be disregarded. Moreover, Wu et al. [27] extracted the laser line by filtering, image segmentation, and binarization. Despite the significant efforts made by many researchers to extract the laser line profile, these presented methods remain inapplicable to a curved laser stripe such as a square groove. These methods are also sensitive to noise, such as welding dust, welding spatter, and strong arc.

The last phase of extracting the weld seam groove is feature recognition. In this phase, the main features of the weld seam groove will be extracted, features that reflect the structural characteristics of the weld seam groove.

These features could be points on the weld seam groove. These locations on weld seam grooves are often at the centerline’s turning point. The distribution and number of these turning points depend on the type of weld seam groove and its structural characteristics. Identifying these turning points in the laser line profile is more complex than it first appears because the skeleton is not a continuous line and may even be intermittent. Additionally, welding noise may make it difficult to identify the turning point’s location. One method for obtaining these feature points is the template-based method [28]. Although this method is effective and used in many research, it might be difficult to use when several turning points occur simultaneously. The image differentiation approach [11] is another way for obtaining the corner points. This method, which is based on the second central difference, identifies the center point of the corner points as the seam center. Muhammad et al. [22] extracted the corner points of V-groove based on pixels’ intensity distribution and neighborhood search. Wu et al. [27] extracted the corner point of the V-groove based on the highest intensity of light projected on the weld seam groove. Li et al. [15] extracted the corner points of V-groove by the character point detected by analyzing slope.

To conclude, most of the previously proposed methods for extracting the image features of the weld seam groove focus on the detection of the weld groove, such as narrow butt joints [1,2,3], tee-joints [4, 5], lap joints [6], and V-joints [7,8,9], and research on the detection of the square-groove is scarce. In addition, some of these proposed methods are sensitive to the light noise induced by various disturbances near or in the seam region. Moreover, most research publications focus on identifying and extracting the weld seam groove’s corner points rather than depth calculation and value comparison.

To overcome the problems mentioned above, this paper presents a new method for extracting the square groove’s feature points, based on Förstner [29] with a Canny edge [30] operator using a laser vision sensor. Therefore, the purpose of this paper is to address the feature extraction algorithm of the square weld groove to make weld seam recognition more accurate.

2 Methodology

2.1 Overview

The features of the square groove are extracted in three steps, as shown in Fig. 1. The first step is the calibration of the laser vision sensor, where the image is captured from a CCD camera at the top of the piecework. The second step is the extraction of the middle line of the laser line profile of the square groove, which is based on the Canny edge operator and Haralicks facet model. The next step is the extraction of the square groove’s corner points, which is based on the Förstner algorithm. Förstner point detection extracts significant points from an image, where significant points are points that differ from their neighborhood. Figure 1 shows the schematic diagram for the laser vision sensor. The CCD camera will scan the workpiece once the laser lights have been projected onto it; then, it will send the image to the PC for image processing. Förstner with Canny edge operators will be the basis for weld seam recognition, which also involves the extraction of weld image features.

Fig. 1
figure 1

Structure of laser vision sensor

Full size image

2.2 Calibration of laser vision sensor

One of the key components for accurately extracting image features is the calibration of the laser with the vision sensor. The calibration of the laser vision sensor is based on the laser triangulation method [31], as shown in Fig. 2.

Fig. 2
figure 2

Structure of laser vision sensor

Full size image

The basis of laser triangulation is a non-contact measuring system. It is used to calculate the distance between points on an object’s surface and a reference plane [32]. After the laser lights will strike the workpiece, the light will be scattered and recorded by a detector [32].

The perpendicular projecting-oblique receiving structure is the type of physical structure of the laser vision sensor in this paper. Perpendicular projecting oblique receiving structure is selected because the structure is simple and the angle between the laser and camera can be manageable [33].

In Fig. 2, the center axes of the single-laser-structured light emitters and an industrial CCD camera are coplanar. The single-line laser is projected on the workpiece and welding groove, which are mutually perpendicular to the workpiece. The industrial CCD camera captures the stripes of the laser on the workpiece. The angle between the laser vision sensor and the workpiece is estimated to be (30°).

In this research, a single-line laser vision sensor is chosen and used, and its characteristics are provided in Table 1.

Table 1 Characteristics of laser
Full size table

There are many parameters for accurately extracting the image features of the weld seam groove. One of the parameters is the type and the quality of the camera. The system is more costly with a high-quality camera. Therefore, robust image processing techniques will reduce costs [13]. A DFK 72AUC02 color camera is selected in this paper, and Table 2 shows the characteristics of the camera.

Table 2 Characteristics of camera
Full size table

There are many types of joints in butt welding joint recognition in industries but this research is focused only on butt joints welding types with the workpiece 4-inch × 4 inches with thicknesses of 3.9 and 6 mm. Mild steel will be the material of choice because it is neither too light nor too dark to be suitable for metal inert gas (MIG) welding. The shape of the butt welding joint in this research is a square groove. Table 3 shows the characteristics of the square weld seam.

Table 3 Characteristics of square weld seam
Full size table

2.3 Central line extraction

In this phase, the central line of laser stripe of the square groove will be extracted. The process of extracting the central line is through three stages, (1) image preprocessing; (2) laser line recognition; (3) central line of the laser stripe. The steps for extracting the central line of laser stripe of the square groove in the image are shown in Fig. 3.

Fig. 3
figure 3

Image processing and laser line recognition procedure

Full size image

After acquiring the weld image by the laser vision sensor, the region of interest (ROI) with n columns and m rows will be applied. This is crucial for lowering the amount of information that must be processed and increasing processing effectiveness [11]. Therefore, the ROI is chosen carefully to reduce the processing time.

The filtering techniques will be used once the RIO has been applied. In order to reduce noise in an arc environment, it has been discovered that the median filter is the most effective filter [34]. This comes after Wu et al. (2014) [34] compared the Wiener, median, and Gaussian filtering techniques in order to identify the best filtering method for eradicating noise from the weld image. Therefore, this paper uses median filters [35] as a technique to remove the noise. The median filter is the most well-known order-statistics filter, and it replaces a pixel’s value with the median of the gray levels in its vicinity, as shown below:

$$f^\wedge(x,\;y)=\mathrm{median}(s,\;t)\in S_{xy}g(s,\;t)$$
(1)

In comparison to linear smoothing filters of comparable size, the median filter has far better noise reduction capabilities for a variety of random noise types and produces significantly less blur [6, 7]. In this paper, a 3 × 3 sampling window is used. The steps for removing the noise using median filter are by arranging the numbers in descending order and selecting the median member, as shown below (Fig. 4).

Fig. 4
figure 4

Median filter mechanism for removing the noise

Full size image

After removing the noise, the laser strip of the weld seam groove can be extracted. Therefore, Canny edge detection algorithm is proposed for laser line recognition [30, 36]. This method is robust and efficient for extracting the laser line profile [37].

Most of the methods used for laser line recognition are sensitive to noise and position change, and cannot identify a curved laser stripe such as a square groove, which affects welding quality. Canny edge detection [30] using Haralicks facet model [38] is used to extract curved laser stripes such as a square groove and extract the laser stripe’s central line. The algorithm’s flow chart is shown in Fig. 5:

Fig. 5
figure 5

Laser line recognition using Canny edge detection with Haralicks facet model

Full size image

Haralicks facet model is used to acquire the midline of the square groove’s laser line profile after extracting the laser line. The Haralicks facet model is an example of a parametric edge detection solution [38]. In principle, this can be used for the detection of edges, regions, and points. This method aims to use a first-order polynomial f to suit local sections of the image surface g (sloped planes or facets). The facet \(f\) is represented by three parameters, \(\alpha , \beta ,\mathrm{ and }d \tau\), which can be evaluated using least squares estimation. The model is given by the following:

$$g(r, c) = \alpha r + \beta c + \tau + n(r, c)$$
(2)

Each pixel has a matching light intensity value that ranges from 0 to 255. 0 refers to the lowest light intensity, while 255 corresponds to the maximum light intensity. To extract the middle line of the laser stripe, the Haralicks facet model will look to the pixels with the maximum highest light intensity and select it.

2.4 Corner point recognition

Based on the centerline of the laser stripe, the feature points of the square groove are extracted based on Förstner algorithm [29]. In actuality, extracting the features points of the weld seam groove presents numerous challenges. One of the challenges is that the laser stripe extracted is not in its perfect shape. The extracted laser stripe may have discontinuities along the lines, and noise may cause the feature points to be suppressed to values greater or lower than their true values. This situation makes accurately detecting these points more challenging. In this paper, detecting the corner points of the square groove is based on Förstner algorithm. Förstner algorithm is used for extracting the corner points in many applications. It is a more efficient operator for extracting feature points from images and has a higher noise resistance [39]. Following are the steps to extract feature points using the Förstner algorithm [39].

The equations of the Robert gradient are as follows:

$${g}_{v}=g\left(x+1,y\right)-g\left(x,y+1\right)$$
(3)
$${g}_{u}=g(x+1,y+1)-g(x,y)$$
(4)

where g(x + 1,y + 1) stands for the gray value of (x, y + 1) and \(({g}_{u},{g}_{v})\) signifies Robert’s gradient operator in the corresponding directions.

The following formula is used to calculate each feature point’s covariance matrix, or cov:

$$\mathrm{cov}=\left[\begin{array}{cc}\sum {g}_{u}^{2}& \sum {g}_{u}{g}_{v}\\ \sum {g}_{v}{g}_{u}& \sum {g}_{u}^{2}\end{array}\right]$$
(5)

Formulas (6) and (7) are used to determine each point’s weight \(\omega\) and roundness \(R\).

$$\omega =\frac{\mathrm{det}\left(\mathrm{cov}\right)}{\mathrm{trace}\left(\mathrm{cov}\right)}$$
(6)
$$R=\frac{4\mathrm{det}(\mathrm{cov})}{(\mathrm{trace}(\mathrm{cov}){)}^{2}}$$
(7)

where trace (“cov”)) is the covariance matrix’s trace and det(cov) is the covariance matrix’s determinant (cov). Finally, alternative points are determined by comparing weight \(\omega\) and roundness \(R\) with the provided threshold.

3 Experimental results and discussion

An experimental test platform is developed once the laser vision sensor, depicted in Fig. 6, has been designed. The purpose of this experimental test is to evaluate the viability of the proposed approach as well as the logic of the laser vision sensor’s structural design. Therefore, a series of experiment tests is carried out to extract the square groove’s feature points (Fig. 7).

Fig. 6
figure 6

Corner point recognition of square groove using Förstner algorithm

Full size image
Fig. 7
figure 7

Laser vision sensor projection of workpiece

Full size image

The workpiece is built of mild steel and has thicknesses of 6 mm and 3.9 mm. The laser vision sensor also includes a single-line laser and a CCD camera on a platform. The physical design of the laser vision sensor is calibrated with the workpiece by utilizing the principle of laser triangulation. The laser and camera make a 30° angle with one another. A 45-cm gap separates the workpiece from the laser vision sensor (LVS). In a well-lit setting, the experiments were carried out. On the tabletop, the workpiece was positioned in the center. The laser vision sensor is perpendicular to the workpiece. The shape of the weld seam is a square weld groove.

3.1 Recognition of the centerline of the square groove

The steps of acquiring the information about the square weld groove start with extracting the central laser line. The positional and geometrical features of the square weld groove, including the corner points, will be extracted once the center line of the laser has been extracted. The resulting of extraction of middle line of the square groove is shown in Fig. 8.

Fig. 8
figure 8

Middle line recognition of the laser stripe of the square groove by Canny edge detector with Haralicks facet model

Full size image

A Canny edge detector with Haralicks facet model was used for extracting laser centerline of the square groove. The Halcon software was used to preprocess images and recognize laser centerlines. Due to the fact that not all of the picture’s pixels are important, the obtained digital grayscale image may be thought of as a pixel array. To minimize computation costs, a ROI of m rows and n columns of pixels was chosen before further processing, as shown in Fig. 8. Following the selection of the ROI, a median filter with a size of 3 × 3 was used, and its ability to eliminate noise is successfully shown in Fig. 8.

After applying the median filter, the laser line profile was extracted using Canny edge detection. Since the laser stripe region has the highest light intensity, the Canny edge detector will search for the highest light intensity with m rows and n columns of pixels. Based on the result in Fig. 8, the extracted laser line profile of the square groove is a collection of pixels with the highest light intensity, representing the laser stripe that follows the contour of the square groove.

Next, the laser stripe’s center line was extracted using the Haralicks facet model and a matching value of the light intensity for each pixel ranging from 0 to 255. With m rows and n columns of pixels, the Haralicks facet model looks for the maximum light intensity. The numbers 0 and 255 represent the lowest and maximum light intensities, respectively. Figure 8 shows that the central laser line is recognized, which proves the effectiveness of this method. Therefore, the Canny edge detector with the Haralicks facet model accurately extracted the central line of the laser stipe of the square groove, as illustrated in Fig. 8.

3.2 Feature information extraction of square groove

The turning points at the edge of the weld seam groove are always regarded as feature points in welding applications. To get the location and geometry details of the weld joint, they must be extracted.

Therefore, in this paper, the Förstner method was used to extract the corner points of the square groove. In Fig. 9, the result of Förstner’s algorithm’s corner point recognition is shown.

Fig. 9
figure 9

Corner point recognition by Förstner algorithm

Full size image

Förstner algorithm extracts significant points from an image, as illustrated in Fig. 9. Significant points are those distinct from their surroundings, that is, those where the picture function varies in two dimensions. These variations occur at the intersections of image edges, known as junction points, and in areas where color or brightness vary from the surrounding neighborhood, known as area points.

The precise coordinates of the square groove weld feature points can represent the straight line’s junction point. Point 1’s coordinate is (332, 249), Point 2’s coordinate is (343, 389), Point 3’s coordinate is (375, 247), and the coordinate of the point is (382, 388). Therefore, the following are the distinctive characteristics of the square-shaped weld: There are 43 pixels of height and 140 pixels of width.

In terms of accuracy, Förstner [29] and Harris [40] algorithms are compared. The Förstner algorithm is compared with the Harris algorithm to assess which one is more accurate in extracting the square groove’s features points and less sensitive to noise, as demonstrated in the experimental results in Figs. 10 and 11. The results of the comparison between the Förstner and Harris algorithms are shown in Figs. 10 and 11.

Fig. 10
figure 10

Corner point recognition by Harris algorithm with Laplacian of Gaussian and Haralicks facet model

Full size image
Fig. 11
figure 11

Corner point recognition by Förstner algorithm with Canny edge detection and Haralicks facet model

Full size image

In this research, we compared the Harris algorithms with our proposed method to evaluate which one has high accuracy in extracting the image features of the square groove. Therefore, width and thickness measurements are taken continuously, and the findings are recorded and shown in Figs. 12 and 13. The measurement results showed that the measured thickness by the Förstner algorithm matches the actual value, whereas the measured thickness by the Harris algorithm has some errors.

Fig. 12
figure 12

Measurement of thickness

Full size image
Fig. 13
figure 13

Measurement of width

Full size image

According to the aforementioned experimental findings, Table 4 shows that the Förstner algorithm is more accurate than the Harris algorithm in extracting the feature points of the square groove.

Table 4 Accuracy of corner point extraction algorithm
Full size table

4 Conclusion

This paper introduced a new method for extracting the feature points of the square groove, which is based on Förstner with a Canny edge operator using a laser vision sensor. Because the process of acquiring the weld image is difficult, there is a lot of noise in the final product. Images must be processed using accurate and reliable procedures. The proposed method can accurately detect the feature extraction of the square welding groove. The outcomes showed that the proposed method can accurately extract the laser stripe’s center line and the corners of the square groove. This paper’s findings have the following advantages:

  1. (1)

    Through Canny edge detection with Haralicks facet model, the centerline of the laser stripe can be extracted without any discontinuity, which demonstrates this method’s ability for extracting curved groove without any discontinuity such as square groove.

  2. (2)

    To extract weld feature points of the square weld groove, the Förstner algorithm, which has good resistance to weld noise and can meet the high accuracy criteria of weld identification, is proposed.