Keywords

1 Introduction

Tube units have several parameters that are controlled by optoelectronic sensors for linear and angular displacement of components. These sensors ensure the relative positioning and alignment of the construction elements during assembly and processing [1, 2]. In some cases, optoelectronic sensors for spatial position control (displacement) of turbo unit components are used as part of a comprehensive system for monitoring these parameters. However, systems incorporating optoelectronic sensors have their drawbacks. Such systems include an excessive number of system elements, which reduces reliability, and manual calibration is required before using these optical systems [3]. The number of elements can be significantly reduced, and the calibration process simplified, by utilizing computer vision systems that employ visual feedback (VF) in the form of visual servo control. The goal of visual servo control is to minimize the error between the desired position and the visual data [4].

Visual servo control methods are divided into two types: image-based visual servo (IBVS) and position-based visual servo (PBVS). Each method has its advantages and disadvantages. IBVS uses distinctive image features (points [5], lines [6], apertures [7]) as the feedback signal. The main advantage of IBVS is that it relies solely on image data without requiring computation of the spatial coordinates of the controlled object. However, IBVS also has its drawbacks and is sensitive to control inputs. A large initial error can lead to overshooting, causing the controlled position to exit the image and resulting in algorithm failure. Moreover, the generated motion trajectories can be physically incorrect or suboptimal. These limitations are due to nonlinearities associated with the transformation between image and spatial coordinates [8]. These drawbacks can be overcome by PBVS, where the controlled position is represented in spatial coordinates. This coordinate transformation from image to spatial allows for stable algorithm performance and avoids the issues associated with IBVS.

When choosing computer vision systems, attention should also be given to the specific application domain. For instance, when working with pipes that have a homogeneous, non-textured surface, passive computer vision systems such as stereovision cannot function properly. To ensure reliable visual characteristics for weakly textured objects, many researchers have used laser emitters (structured light) as the visual feedback signal in visual servo control [9, 10]. In the experiments conducted in this study, precise positioning was achieved through reliable visual servo control based on structured light and a camera integrated into the computer vision system. However, when dealing with pipes of a large diameter, computer vision systems that include a standard camera are not capable of functioning properly due to the laser beam extending beyond the camera field of view (see Fig. 1). Therefore, the integration of a wide-angle camera (FishEye camera) is necessary (see Fig. 1).

Fig. 1.
figure 1

Left image - standard camera with a viewing angle of 70°. Right image - FishEye camera with a viewing angle of 180°.

Full size image

The objective of this article is to optimize methods for aligning pipes based on a computer vision system in order to achieve high positioning accuracy. Since the surface of steel pipes lacks texture, the proposed computer vision system incorporates structured light, which is a suitable method for obtaining an accurate surface profile of the pipes. One distinctive feature of the proposed system is the integration of a fisheye camera, allowing for measurements of pipes with large diameters. The article discusses the use of the Visual Servoing PBVS method in the proposed measurement system, which controls the position of the pipe’s center relative to a given value. Additionally, the article covers an adapted technology for calibrating the computer vision system.

2 Computer Vision Model

2.1 Model Description

The computer vision system considered in this research consists of a panoramic camera and a structured light source (see Fig. 2). The camera, integrated into the setup, has a field of view of 180°, with the potential to expand up to 210° and 240°. The distance between the laser plane and the camera can be adjusted using a simulator. Additionally, the orientation of both the camera and the laser plane can be modified. The lunette serves as the executive component of the tracking system. It comprises a housing, a pipe centering device, and two actuators for horizontal and vertical movements (Fig. 2). The lunette’s electric motors continuously correct the pipe’s position based on feedback signals to ensure that the center of the inner surface remains aligned with its axis. The center of the inner surface is determined by projecting the extracted laser points onto the inner surface of the pipe (Fig. 2). Subsequent paragraphs, however, are indented.

Fig. 2.
figure 2

Description of the computer vision system. The left image shows an overall view of the system. The right image displays a snapshot from the fisheye camera.

Full size image

The described system is a tracking system. The electric drive is considered near the desired position when the angle control error ∆α = αin −α≈0 (see Fig. 3). In this structural diagram, the position control loop consists of the following components in the direct channel: PC (corresponding to the position controller PC and the measurement and signal transformation elements for position error on the shaft of the working mechanism), SRC (corresponding to the closed-loop speed control), and the integrator I (which accounts for the conversion of the output shaft’s angular velocity into an angle). In the feedback channel, there is the computer vision sensor S (which measures the current position of the working mechanism’s shaft).

Fig. 3.
figure 3

Structural diagram of a positional electric drive.

Full size image

2.2 Computer Vision Sensor Model

The camera model was previously described in [11]. Taking this information into account, the equation for projecting the laser plane can be written as follows:

$$\left[\begin{array}{c}\begin{array}{c}u\\ v\end{array}\\ \left.f(\rho \right)\end{array}\right]\times \left[\begin{array}{ccc}{r}_{1}^{c}& {r}_{2}^{c}& {r}_{3}^{c}\end{array}\right]\left[\begin{array}{ccc}{r}_{1}^{l}& {r}_{2}^{l}& \begin{array}{cc}{r}_{3}^{l}& {t}^{l}\end{array}\end{array}\right]\left[\begin{array}{c}X\\ Y\\ \begin{array}{c}Z\\ 1\end{array}\end{array}\right]=0$$
(1)

where u, v are the pixel coordinates of the image; \(\begin{array}{ccc}{r}_{1}^{c}& {r}_{2}^{c}& {r}_{3}^{c}\end{array}\) are the vectors of the camera rotation matrix; parameters \(\begin{array}{ccc}{r}_{1}^{l}& {r}_{2}^{l}& \begin{array}{cc}{r}_{3}^{l}& {t}^{l}\end{array}\end{array}\) represent the laser plane transition matrix. The polynomial \(f\left(\rho \right)\) can be expanded as:

$${f\left(\rho \right)=a}_{0}+{a}_{2}{\rho }^{2}+\cdots +{a}_{N}{\rho }^{N}$$
(2)
$$\rho = \sqrt{{\left(u-{u}_{c}\right)}^{2}+{\left(v-{v}_{c}\right)}^{2}}$$
(3)

where \({a}_{i}\) are coefficients; N is the degree of the polynomial; \({u}_{c}\), \({v}_{c}\) are the coordinates of the image center.

The laser plane is positioned at a fixed distance from the optical center of the camera along the Z-axis in Fig. 2. This distance corresponds to the third row of the vector \({t}^{l}\). Therefore, the global coordinates along the Z-axis remain unchanged. Thus, in Eq. (1), Z = 0, and this equation can be transformed into:

$$\left[\begin{array}{c}\begin{array}{c}u\\ v\end{array}\\ \left.f(\rho \right)\end{array}\right]\times \left[\begin{array}{ccc}{r}_{1}^{c}& {r}_{2}^{c}& {r}_{3}^{c}\end{array}\right]\left[\begin{array}{ccc}{r}_{1}^{l}& {r}_{2}^{l}& {t}^{l}\end{array}\right]\left[\begin{array}{c}X\\ Y\\ 1\end{array}\right]=0$$
(4)

3 Calibration

Problem Statement: We have a computer vision system consisting of a fisheye camera and a structured light source, capable of measuring the depth of the surrounding space. The object under control is the center of a pipe. To monitor the position of the pipe’s center relative to a specified value, it is necessary to align the optical axis of the computer vision system with the axis of the pipe’s inner surface.

Solution: To solve the given problem, we performed a modification of the calibration target previously introduced in [14, 15]. The modification involved adding plungers to the edges of the target (see Fig. 4). The plungers, by pressing against the inner surface of the pipe, evenly distribute the applied force, aligning the axis of the calibration target with the axis of the pipe’s inner surface. Subsequently, using the calibration procedure described in [11], we can align the optical axis of the computer vision system with the axis of the pipe’s inner surface.

Fig. 4.
figure 4

Calibration target. The left image is a general view of the system. Right image - fisheye cameras.

Full size image

Experiment: Below is a visual example of a measurement to demonstrate the importance of the calibration process. Before the experiment, the camera and the laser plane were rotated arbitrarily. After that, the vision system was calibrated according to the principle described in [14]. On Fig. 5 and Table 1 compare the pipe inside diameter measurements.

Fig. 5.
figure 5

Left to right: input image; extracted laser dots; projection of laser points into spatial coordinates (projection of a calibrated system - blue color, projection of an uncalibrated system - magenta color).

Full size image
Table 1. Evaluation of measurement results
Full size table

According to Fig. 5 it can be seen that in the case of a calibrated system, the projection of laser dots on the inner diameter of the pipe has the proper shape and the camera axis is aligned with the axis of the inner surface of the pipe, coordinates (0, 0). Table 1 shows that in the case of a calibrated system, the resulting deviation of the measured value of the pipe diameter is within the allowable limits, while, for an uncalibrated system, the measured value significantly exceeds the allowable limits.

4 Visual Servo Control

Problem statement: there is a vision system (fisheye camera and structured light source), with the ability to measure the depth of the surrounding space; a controlled object (pipe center) observed in two states (current and desired) and a control system (servomotors) that moves the pipe in vertical and horizontal positions. Using the calibration parameters of the vision system, it is possible to restore three-dimensional information about the pipe section and determine the spatial coordinates of its center. Thus, the position of the controlled center of the pipe can be characterized by a set of spatial points (X, Y, Z), where (X0, Y0, Z0) is the desired position of the center, and (X1, Y1, Z1) is the current position of the center. The research problem is to minimize the error between the current and desired positions.

Solution: to solve this problem, a model of this technological process was developed in the MATLAB mathematical modeling system and a virtual model of the complex in the Unity software environment, where communication between programs is carried out using the TCP/IP protocol. The topology of the system is shown in Fig. 6.

Fig. 6.
figure 6

System topology

Full size image

The goal of the traditional visual servo control scheme is to minimize the error e(t), which is defined as e(t) = p(t) – r, where p(t) is the current position of the pipe center and r is the desired position. To generate a control signal for the servo drives (u) that move the pipe in the vertical and horizontal directions, a PID controller was chosen. Further, the laser point (l) on the inner surface of the pipe is extracted by the vision system. After that, the current position of the pipe (p) is calculated.

Experiment: The experiments included in this section were aimed at evaluating the possibility of positioning the center of the pipe in the desired position, using the proposed vision system, as well as testing the stability of this control method under disturbing influences. The initial configuration of the vision system is as follows: the optical center of the camera has coordinates (0, 0, 0); laser emitter, has coordinates (0, 0, 666) mm. To test the operability of the visual servo control, the pipe at the initial moment of time was moved in the vertical and horizontal directions with coordinates (−300, −250, 666) mm. After that, in the steady state, a distribution was applied, shifting the center of the pipe to a position with coordinates (200, −150, 666) mm. Below are the results.

Figure 7 shows the visual servo control results. Figure 7(c) shows the chronology of the spatial displacement of the center, at the initial moment of time and during the development of the disturbing effect. The figures show that the proposed vision system, using the method of visual servo control, is able to control the position of the center of the pipe.

Fig. 7.
figure 7

(a) Alignment in the horizontal direction. (b) Alignment in the vertical direction. (c) Alignment a general view of error minimization in the horizontal and vertical directions.

Full size image

5 Conclusion

This article presents a solution to the classical problem of pipe centering through the integration of new measurement systems and measurement methods. The proposed vision system, compared to existing optical pipe centering systems, contains fewer electrical elements and mechanical connections, which in turn increases system reliability. Due to the low-textured surface of the pipe, structured light projection onto the inner section of the pipe was used, which significantly simplified image processing and extraction of the necessary information. The fisheye camera, with its wide field of view, allowed for working with pipes of larger diameters. To minimize the error between the desired and current positions, a PVBS controller was chosen, which, in combination with the proposed measurement system, demonstrated its effectiveness during the conducted experiments.

The article also touched upon the importance of calibration of the measurement system. For this purpose, a calibration target was developed to align the axis of the vision system with the axis of the inner surface of the pipe. Calibration of the system allowed for measuring the pipe cross-section within the acceptable limits defined by the technical specifications.

The mentioned results were achieved using the developed virtual environment. In real experiments, there is measurement uncertainty, which complicates the comparison of methods. Moreover, in real conditions, it is difficult or impossible to estimate certain parameters, such as the position, orientation of the laser plane, or the value of the semi-cylinder radius, whereas in the simulation environment, they are known.