An Improved Monocular-Vision-Based Method for the Pose Measurement of the Disc Cutter Holder of Shield Machine

Abstract

The manual disc cutter changing of the shield machine has greater security hidden danger, which leads to the urgent need to develop a disc cutter changing robot to replace the over-worn disc cutters. The rotation positioning error of the large diameter cutter head, the motion control error of the disc cutter changing robot, and the complexity of the disc cutter changing bin environment lead to the inaccuracy of the actual pose of the disc cutter. Therefore, it is necessary to increase the machine vision measurement system to accurately measure the pose of the disc cutter or its holder. This paper proposes a cutter holder positioning method, which uses the parallel line features and actual contour area of the cutter holder to determine its initial pose, and then uses the initial pose to calculates its accurate pose based on the distance matching method. The innovation of the method is that the pose measurement is carried out through the unique characteristics of the cutter holder, and it combines the initial pose estimation method based on the geometric characteristics of the cutter holder with the distance matching method which depends on the initial pose to solve the cutter holder pose. Simulation analysis shows that the accuracy of the proposed method is close to that of the most advanced PnP algorithms, and its robustness is better than others. Experimental results also verify the high accuracy of the proposed method.

1 Introduction

Shield machine is a kind of semi-automatic basic equipment, but it still needs manual operations in many aspects. Due to the longtime tunneling operation of the shield tunneling machine, the disc cutters on the cutter head are easy to be seriously worn. Under the bad working environment of high pressure and high humidity, the disc cutters still need to be replaced manually at this stage [1]. The complex structure in the tunnel and many uncontrollable factors lead to many casualties caused by manual disc cutter changing at home and abroad, thus it is very necessary to replace the disc cutter with a robot [2]. To ensure that the robot can accurately grasp the disc cutter for change operation, the machine vision measurement system should have high accuracy and robustness.

Visual measurement systems can locate the object by extracting its features or by artificial markers fixed on it, because of the complexity of fixing artificial markers, they are not considered when the object has obvious features [3, 4]. The coplanar characteristic line segments and characteristic corners of the cutter holder are used to estimate its pose in this paper. The vision measurement system is fixed on the end actuator of the disc cutter changing robot, to ensure the low load and dexterous operation of the manipulator, the monocular measurement method with simple structure and relatively low cost is given priority. At present, there are many pose estimation algorithms based on monocular vision, the perspectives-n-point problem (PnP) algorithm is one of them, which aims to estimate the rotation and translation of a calibrated perspective camera by using n known 3D reference points and their corresponding 2D image projection. Direct linear transformation (DLT) is a classic PnP algorithm with high efficiency but poor robustness due to noise interference [5]. EPnP is a non-iterative algorithm, which uses virtual points to indirectly solve the pose parameters of the target, thus reduces the computational complexity of the algorithm, however, when the number of feature points is small, its solution accuracy and robustness are relatively low [6]. RPnP algorithm can present stable calculation results when there are fewer and more feature points, but it cannot get the unique solution when using least squares error, and its solving accuracy needs to be improved [7]. The basic idea of OPnP and ASPnP algorithm is to formulate the PnP problem into a functional minimization problem and retrieve all its stationary points by using the Gröbner basis technique, these two algorithms have high accuracy and robustness, and are the best PnP algorithms so far [8, 9]. These PnP algorithms have great differences in solving accuracy when the image noise is small and large. Because of the actual situation that the environment of disc cutter changing site is execrable and cutter holder images collected contain strong noise, a method with better robustness is needed to solve the pose of the cutter holder.

The distance matching (DM) method proposed in this paper can solve the pose of the cutter holder without the one-to-one correspondence between the 3D reference points and the 2D image points. Rough positioning is carried out according to the geometric features of the cutter holder, and then its exact pose is obtained by distance matching. In the initial positioning, the 6DOF of the cutter holder pose are solved respectively, the three rotational components are solved by the vanishing points obtained from the parallel lines on the cutter holder surface. The depth information is obtained by the ratio of the inside contour of the cutter holder in the image to its area in the real space, which is used to calculate the other two translational components. The cutter holder pose is accurately solved by the distance matching method, the distance between the collected image of the cutter holder and the template in the established template library is calculated, then the template corresponding to the minimum distance is selected, and its corresponding pose is used as the final estimated pose of the cutter holder.

This paper is organized as follows. Section 2 states the pose estimation algorithm of the cutter holder we propose. In Sect. 3, simulations and experiments are implemented to test the performance of the proposed algorithm. And Sect. 4 gives conclusions.

2 Mathematical Model and Method

2.1 Image Processing

Salient features of the cutter holder are extracted to accurately estimate its pose, such as lines, corners, circles, etc., which are the most commonly used geometric information in the field of robot visual positioning. It is difficult to extract the feature corners and lines directly because of the wear and corrosion on the surface of the cutter holder caused by the long time working of shield tunneling, thus we can extract the overall contour of the cutter holder image, and then obtain the salient features from it to measure the pose. Given the obvious and numerous linear features of the inside contour of the cutter holder, the characteristic straight lines on its surface are extracted for pose calculation.

The detailed algorithm for feature lines detection is summarized in the form of a block diagram in Fig. 1. The whole process of image processing is divided into two blocks: preprocessing block and feature lines extraction block.

Fig. 1.

Image processing flow diagram.

In the preprocessing block, the grayscale images of the cutter holder collected are processed by median filtering using a 7*7 square mask. The template matching method based on grayscale is applied to extract the region of interest (ROI) containing the whole part of the inside contour of the cutter holder from the filtered image. The global threshold segmentation method is used to segment the ROI by setting the threshold range from 70 to 255, and the binary image of the cutter holder is obtained. In the feature lines extraction block, many closed contours are obtained by edge extraction of the binary image, and the closed inside contour of the cutter holder with the longest length is selected by length sorting. Based on the curvature constraint, a straight line or arc with a length of at least 6 pixels is used to approximate the inside contour, to segment it into several line segments. The line segments whose length is smaller than the threshold, which is set as 50, are eliminated from the obtained line segments, and the remaining segments are collinearly processed to obtain the characteristic segments, which are used to solve the pose of the cutter holder.

2.2 Initial Pose Estimation

The initial pose estimation method proposed in this paper calculates the rotation and translation components of the cutter holder pose in turn. The rotation components of the cutter holder pose are obtained by using the extracted parallel characteristic line segments. The depth information is obtained by the ratio of the area of the inside contour of the cutter holder in the image to its area in space, which is helpful to calculate the remaining two translation components.

Through the above image processing method, 10 parallel lines are obtained in the horizontal and vertical directions of the cutter holder, P₁P₂ and P₄P₃, P₄P₁ and P₃P₂ in Fig. 2 are the two pairs of parallel lines arbitrarily selected from them. A pair of parallel lines in space projected onto the image plane have one and only one intersection point, which is called the vanishing point. P is the origin of the world coordinate system, the image point P_j corresponding to the space point P_ij, and the values of i and j are from 1 to 4. The vanishing point between parallel lines P₁P₂ and P₄P₃ is represented by P_v1, while the vanishing point between P₄P₁ and P₃P₂ is represented by P_v2. The image coordinates of P_v1 and P_v2 are calculated according to the characteristic line segments obtained from the above image processing. A pair of parallel lines can get a vanishing point, according to the obtained cutter holder of many pairs of parallel lines can get some vanishing points. The image point with the smallest sum of the distances to all vanishing points is used as the final vanishing point for the pose solution. The symbols of the two vanishing points obtained by the solution are also expressed by P_v1 and P_v2.

Fig. 2.

Schematic diagram of pose measurement based on vanishing points.

Based on the image coordinates of P_v1 and P_v2, the imaging point coordinates normalized to the image plane at the camera's focal length can be calculated as follows:

$$ \left[ {\begin{array}{*{20}c} {x_{1cvi} } \\ {y_{1cvi} } \\ 1 \\ \end{array} } \right] = \left[ {\begin{array}{*{20}c} {k_{x} } & 0 & {u_{0} } \\ 0 & {k_{y} } & {v_{0} } \\ 0 & 0 & 1 \\ \end{array} } \right]^{{{ - }1}} \left[ {\begin{array}{*{20}c} {u_{vi} } \\ {v_{vi} } \\ 1 \\ \end{array} } \right] $$

(1)

In formula (1), (u _vi, v _vi) is the image coordinate of the vanishing point P_v1, (x_1cvi, y_1cvi, 1) is the coordinate of P_v1 normalized to the image plane at the focal length, where i =1, 2. k _x and k _y are the magnification coefficients of the camera from the imaging plane coordinates to the image coordinates. (u₀, v₀) is the image coordinate of the center of the optical axis of the camera.

Geometrically, the vanishing point of a line in space is obtained by the intersection of a ray that is parallel to the line and passes through the center of the camera and the image plane, so O_CP_v1 is parallel to P₁P₂ and P₄P₃. Therefore, [x_1cv1 y_1cv1 1]^T is both the position vector of P_v1 in the camera coordinate system and the direction vector of the X-axis of the world coordinate system. Similarly, [x_1cv2 y_1cv2 1]^T represents the direction of the Y-axis of the world coordinate system. By normalizing the two vectors into unit vectors, the X-axis and Y-axis components of the rotation matrix R from the world coordinate system to the camera coordinate system can be obtained. The cross-product of the two components is further solved to obtain the Z-axis rotation component of R.

The normal vector $\overrightarrow{n}$ of the plane P₁P₂P₃P₄ is obtained according to R. The image point of P is P₁, a plane with normal vector $\overrightarrow{n}$ passes through P₁, and its intersection points with O_CP_i1, O_CP_i2, O_CP_i3, and O_CP_i4 are P_j1, P_j2, P_j3, and P_j4 respectively, thus the lines P₁P₂, P₄P₃, P₁P₄, and P₂P₃ are parallel to P_j1P_j2, P_j4P_j3, P_j1P_j4, and P_j2P_j3 respectively. According to the image coordinates of P_j1, P_j2, P_j3, and P_j4, the area of the polygon P_j1P_j2P_j3P_j4 and the distance d₁ from point O_C to it are obtained. Using the principle of similarity, the square of the ratio of d₁ to the distance d₂ from O_C to the plane P₁P₂P₃P₄ is equal to the area ratio of polygon P_j1P_j2P_j3P_j4 to the polygon P₁P₂P₃P₄, thus the value of d₂ can be solved, which represents the depth information of the cutter holder. The coordinate of P in the camera coordinate system is also obtained according to the similarity principle, so as to obtain the remaining two translational components of the cutter holder.

2.3 The Proposed Distance Matching Method

The distance matching method takes the initial pose as input and can solve the local optimal solution. Substituting the camera internal parameter matrix obtained from the monocular camera calibration, the pose template library of the cutter holder is established through the projection model of the camera. The distance between the collected cutter holder image and the template in the library is solved and taking the pose of the template corresponding to the minimum distance as the final estimated pose of the cutter holder. The flow chart of the distance matching method proposed in this paper is shown in Fig. 3.

Fig. 3.

Flowchart of distance matching method.

The origin of the workpiece coordinate system of the cutter holder is at the center of its surface, the Z-axis is perpendicular to its surface, and the X-axis and Y-axis are respectively parallel to the long and short sides of its outer rectangular contour. The workpiece coordinate system is set as the world coordinate system, then the relative pose between the world coordinate system and the camera coordinate system is the final required cutter holder pose. To meet the requirement that the grasping accuracy of the manipulator should reach 1 mm, the position moving step and the rotation step is set as 0.1 mm and 0.1° respectively to transform the cutter holder pose within the known relative motion range between it and the manipulator. We can get the image coordinates corresponding to the spatial feature corner points in each pose state of the cutter holder by using the camera projection model. The pose template library of the cutter holder is constructed, each template corresponds to a pose of the cutter holder in the camera coordinate system, and contains the image coordinates of 20 characteristic corner points on the surface of the cutter holder. Due to a large number of templates, compared with the way of storing images directly, the storage capacity of the template in the form of image coordinates of feature points is greatly reduced. Although it takes a long time to establish the pose template library, it is an offline operation and does not occupy the time of real-time pose estimation.

The operation converting a binary image to an approximate distance image is called a distance transformation (DT), which can produce a grayscale image, the grayscale value of each pixel in it is the distance between the pixel and the nearest background pixel. The image processing method described above is applied to the collected cutter holder image to obtain 20 feature line segments, and 20 feature corners are obtained by intersecting the extension lines of the segments. The distance transformation diagram of the binary image containing only 20 feature corners is obtained through distance transformation, as shown in Fig. 4, in which the gray value of each pixel is expressed as the distance between the pixel point and the nearest feature corner point. We can substitute the image coordinates of the characteristic corners of any template in the pose template library into the distance transformation graph to get 20 distance values, and solve their root mean square value as the distance between the actual cutter holder and the template. The pose of the template corresponding to the minimum distance in the library is selected as the estimated pose of the actual cutter holder.

Fig. 4.

DT image. The larger the distance the lighter the tone.

By changing the step size from large to small, the efficiency of searching the template corresponding to the minimum distance in the template library is improved. The initial step sizes of translation and rotation directions are set as 5 mm and 5° respectively, and the template corresponding to the minimum distance is searched from the library. The pose corresponding to the obtained template is taken as the initial pose, and the step size of translation and rotation direction is reduced to 1 mm and 1° for local search to obtain the estimated pose. The above process is repeated with the step sizes of 0.5 mm and 0.5°, 0.2 mm and 0.2°, 0.1 mm and 0.1° in the direction of translation and rotation in turn, and the template pose corresponding to the final minimum distance is taken as the estimated pose of the cutter holder.

3 Experimental Results

3.1 Experiments with Synthetic Data

The pose estimation method proposed in this paper is tested by simulation experiments. In order to accurately evaluate the accuracy and robustness of the algorithm, the experiment will be compared with various mainstream PnP algorithms, including ASPnP, DLT, EPnP, OPnP and RPnP. Given a virtual perspective camera with image size 4112 × 3008 pixels, pixel size 3.45 µm, and focal length 8.5 mm, the cutter holder pose is generated randomly, and the rotation and translation distribution ranges are [−60°, 60°] and [−90, 90] mm respectively. The feature points on the surface of the cutter holder are projected onto the image plane based on the camera imaging model. Different levels of Gaussian noise are added to the projected image points, and for each noise level, 1,000 test data sets are generated. The pose solving algorithms are used to solve the cutter holder pose, and the error between the calculated pose and the randomly generated pose is obtained.

Due to the harsh environment of the measurement site, the actual cutter holder images collected contain strong noise, this paper mainly studies the influence of noise intensity on the measurement accuracy. Since the features to be extracted are 20 corner points on the surface of the cutter holder, the number of input reference points is fixed as 20, and the variance of input noise is changed from 0 to 5 pixels (the step is 0.5 pixels) to verify the solving accuracy of each algorithm. The DM algorithm proposed in this paper and the other PnP algorithms are implemented in MATLAB and 500 test runs are performed. The error values of each degree of freedom are expressed by the standard deviation of the error values of 500 pose solutions. The error results obtained are shown in Fig. 5.

It can be seen from the simulation results that the accuracy of the RPnP method is lower than that of other PnP methods, followed by the accuracy of EPnP and DLT methods. The accuracy of the OPnP method is the same as that of the ASPnP method, which has the best performance among these PnP methods. With the increase of input noise, the accuracy of the PnP methods decreases sharply. Compared with these PnP methods, when the noise variance is less than 2.5 pixels, the solving accuracy of the DM method is slightly lower than that of the PnP methods. On the contrary, when the noise variance is higher than 2.5 pixels, the accuracy of the DM method is better than that of the PnP methods with the increase of noise. In the case of noise enhancement, the accuracy of the PnP method drops sharply, while the accuracy of the DM method changes little and remains relatively stable, especially in the direction of depth, which indicates that the robustness of the DM method proposed in this paper is better than the PnP method. The high precision and good robustness of the DM method under high noise indicate that it is more suitable for the construction site of disc cutter changing operation with higher noise.

Fig. 5.

Accuracy of all algorithms when noise level changes.

3.2 Experiments with Real Images

To make the method can be applied in practice, the industrial field environment is simulated in the laboratory. The experiment is carried out in a dark environment, the space in the tool changing bin is narrow, and the larger robot has a smaller movement range in the disc cutter changing bin. The motion control precision of the disc cutter changing robot is 5 mm, so the cutter holder can be accurately positioned within the range of [−10, 10] degrees and [−20, 20] millimeters in the direction of rotation and movement respectively.

Fig. 6.

The monocular vision positioning system of the cutter holder.

The moving platform as shown in Fig. 6 is used to carry out a positioning test on the cutter holder, the motion accuracy of the platform in each direction of translation is 0.01 mm, and the motion accuracy in the direction of Z-axis rotation is 0.02°. Allied Vision Manta G-1236 camera is used as the image acquisition equipment in the experiment, its resolution, pixel size, and focal length are the same as the analog camera used in the simulation experiment.

Zhang Zhengyou calibration method is used to calibrate and determine the internal parameters of the camera [10], and the 7*7 circular calibration plate used in the calibration is made of alumina panel and glass substrate materials, and its machining accuracy is 0.01 mm. The calibration results are listed in Table 1.

Table 1. The results of camera calibration.

The cutter holder images are collected with the calibration monocular camera, and the cutter holder pose is changed by the moving platform within the range set above. Every time the motion platform changes the pose of the cutter holder, an image of it is collected, and the pose at that time is recorded. There are 100 cutter holder images that are randomly collected, and the pose of 100 groups of the cutter holder is recorded at the same time. The image processing, initial pose estimation, and distance matching method described above are used to solve the pose of the actual cutter holder. The calculated pose is compared with the known actual pose, and the standard deviation of the errors of each degree of freedom is obtained. The PnP algorithms are also used to solve the cutter holder pose, and the error results obtained are shown in Table 2.

Table 2. The standard deviation of pose error.

The cutter holder used in this paper is a new type of cutter holder with high machining accuracy, which has not been put into use in the construction site, so the image noise of the cutter holder collected is small. The experimental results are similar to the simulation results in the case of low noise.

4 Conclusion

In this paper, a pose estimation algorithm is proposed based on the characteristics of the disc cutter holder of the shield machine, which makes use of vanishing point property and similarity principle for initial positioning, and then carries out precise positioning according to the distance matching method. The simulation results show that the proposed algorithm has high accuracy and robustness. The actual experiment proves that the algorithm can achieve high accuracy under low noise conditions. In the future, wear and corrosion treatment will be carried out on the new type of cutter holder to simulate its working state under actual working conditions.