Keywords

1 Introduction

Currently, the use of robots in technological processes has become a model of high-quality industrial automation throughout the world [1]. The robot is a reliable and effective tool for solving various production tasks: loading, packaging, plasma welding, cutting, coating, etc. The technologies of plasma spraying of coatings, as well as the technology of plasma cutting products require accurate observing a number of technological parameters (the distance from the plasma system nozzle to the surface of a workpiece, the nozzle movement speed, etc.) during the entire processing time. Exceeding these parameters beyond the permissible limits can lead not only to defective products, but also to an accident (a short circuit). In cases when the robot program is generated according to a given geometrical model of a finished components or part, the deflection of the shape of the real object from the model often leads to the violation of technological parameters of processing leading to undesirable consequences. This problem is particularly acute in the case of large-size objects, when small relative errors of geometric parameters and object positioning correspond to unacceptably high absolute deviations of the distances between the tools mounted on the manipulator and the object surface. The deeper solution of these problems is pre-scanning the surface of an object.

A modern robot manipulator can be considered as a means of precise position and orientation of an arbitrary instrument. If a distance sensor or a computer vision system element (camera or projector) is used as a tool, the robot manipulator can be an excellent basis for creating a surface scanning system. It should be noted that the tasks associated with the construction of 3D-scanning systems based on robotic manipulators are of considerable independent interest, both practical and theoretical, and the research stimulated by these tasks is carried out both by research units of companies specializing in the manufacture of test and measurement means and means of automation of production, and representatives of the academic environment [2,3,4,5,6,7,8,9].

The motivation for the elaboration of our robotic scanning system was the analysis of the challenges that we faced when using the industrial robot Kawasaki for plasma surface treatment [10,11,12,13], as well as the desire to expand the range of tasks solved by the industrial robots usage. Firstly, we faced the need to applying coatings from biocompatible materials (titanium and alloys based on it, powders of hydroxyapatite, etc.) on the surface of medical products of complex shape. Secondly, the need to cut metal sheets by plasma cutting for small-scale production of large-sized products arises.

The task of reconstructing a surface from a spontaneous point cloud often arises in different engineering applications. We faced the problem of creating a CAD model based on laser-based scanning data of an object, while designing an automatic system of path planning for a robot manipulator. The basic idea of the research is to plan the trajectory based on the scanning data of processed object (finished component).

The main objective of the work is to create a method for automatic reconstruction of accurate and piecewise smooth surfaces from spontaneous 3D points. The segmentation of a point cloud is an important part of constructing a parametric model of an object. The aim of segmentation is partition of an unstructured point cloud into non-intersecting subsets of points. All points within each segment must be close to each other in the selected feature space. The procedure of automatic segmentation proposed by the authors is based on the local analysis of the Gaussian K and mean H curvatures, obtained by applying a non-parametric analytical model.

2 Experimental

An experimental study was conducted with the use of RS-010LA Kawasaki industrial robot (Kawasaki Robotics, Japan). The industrial robot manipulator is a device consisting of movable parts with six degrees of freedom to move to a predetermined profile, managed by a programmable controller E40F-A001. The robot’s arm is equipped either by UPR device (NPPTekhnotron Ltd., Russia) for plasma cutting (Fig. 1), or a MPN-004- microplasmatron (E.O. Paton Institute of Electric Welding, Ukraine) for microplasma spraying powders or wires (Fig. 2). Kawasaki robot is controlled by AS software system. Binocular triangulation laser sensors intended for use in automation systems and non-contact measurement of various geometric parameters are chosen as distance sensors.

Fig. 1.
figure 1

The robot’s arm equipped by UPR device for plasma cutting of metals

Full size image
Fig. 2.
figure 2

Robotic processes of microplasma spraying of powder (a) or wire (b)

Full size image

The KawasakiRS-010LA robot manipulator characteristics are as follows:

  • positioning accuracy - 0.06 mm;

  • maximal linear speed - 13100 mm/s;

  • engagement zone - 1925 mm;

  • working load capacity - 10 kg.

In this study, we focused on describing the development of a 3D-scanning system based on proximity sensors mounted on an industrial robot manipulator. The proposed system can scan an object on the basis of the distance from the surface of an object in a discrete set of points, forming a network with the specified geometric parameters imposed on the surface of the object. A 3D-model of the scanned object is built when using interpolation procedures of grid points. Thus, the elaboration of a system of 3D-scanning requires the generation of a data acquisition system involved in job stacking for the robot manipulator. The system consists in stepping according to scanning points, at each point performing actions, followed by the accumulating the results in the storage system. As precision distance sensors usually have a limited operation range, and some of them require orientation of the axis of the sensor in a direction perpendicular to the area of an object subjected to scanning, it is necessary to elaborate a full-blown system consisting of rough and fine scanning subsystems.

3 Results and Discussion

We have developed a 3D-scanning system, where the process of generating a “point cloud” technically performs the following operations:

  1. (1)

    The scanning route with stop points is specified;

  2. (2)

    The distance to the surface of the scanned object is measured at the stopping point;

  3. (3)

    Scan data is sent to a PC. Communication protocol (data transfer logic) provides the transfer of the current coordinates of the scanner together with the scan data (distance);

  4. (4)

    The obtained data allow us to construct a point cloud and calculate the 3D-coordinates of a set of points on the surface.

The system of rough scanning that uses position sensors with a wide range of measurable distances and is insensitive to the orientation relative to the surface, is used for setting fine scanning sensors in the operating position and preventing accidents during the scanning process. Thus, in order to implement the scanning system we must develop the hardware and software for the data acquisition system briefly described above. The elaboration of the system should include research on testing the algorithms of distance sensors setting in run position and the study of the system specifications in test situations with known geometry. The process of 3D-models creating describes following programming layer of this solution, which encloses kernel software mentioned in previous chapter. This layer consists in Trajectory Planning Block, Surface Generator Block and File Saver. Placing these blocks in the entire system is shown in Fig. 3 and its functions are described later in the text.

Fig. 3.
figure 3

Blocks of solutions of a higher programming layer

Full size image

As shown in Fig. 3, output data from kernel 3D-scanner are in the form of 3D-point cloud. It means that spontaneous set of three-dimensional data is produced and sent into the Surface Generator block, where smooth shaded surface should be generated. As it is known, laser-based scanning is characterized by an automatic method for detecting a point cloud, and at the subsequent stages of processing, human intervention is already required. For 3D-models constructing we have to develop a method of segmentation of the point cloud obtained at the stage of rough scanning of the surface, namely the algorithms of image segmentation and segmentation of the surface based on the calculation of the local geometric properties of the surface. Segmentation splits a complex surface into geometrically homogeneous regions with a simple analytical description.

For surface segmenting we suggest using the variation of the region growing algorithm, called Unseeded Region Growing Algorithm [14]. Although this algorithm was proposed for the segmentation of images, its sufficient commonality allows using it for our purposes. The proposed procedure for the automatic segmentation of the surface is based on local analysis of the Gaussian and mean curvature of the surface obtained when constructing a nonparametric analytical model. In the proposed method, the point cloud is interpreted as a height map, i.e. it is assumed that the discrete function of two variables Z (X, Y) on a grid in the plane XY is preassigned.

In each point, the surface is modeled by a second-order polynomial by the local coordinates. The calculation of the coefficients of the approximating polynomial is carried out considering a selected number of neighboring points lying within a circle of a given radius, for the application of the method of least squares. The main advantage of the nonparametric approach is its commonality: previous assumptions about the local geometry of the surface are not introduced, and, moreover, the analytical model of the surface is not proposed.

Let us consider the approximation of the surface in the local neighborhood of an arbitrary point P with the coordinates \((x_{k},y_{m},z_{km})\) of type (1)

$$\begin{aligned} z_{ij}=c_{0}+c_{1}\cdot (u)+c_{2}\cdot (v)+\frac{1}{2} \cdot c_{3} \cdot u^{2}+c_{4}\cdot u\cdot v+\frac{1}{2}\cdot c_{5\cdot v^{2}}+\varepsilon _{ij} \end{aligned}$$
(1)

where,

$$\begin{aligned} c_{0}=z_{km}, u=x_{i}-x{k}, v=y_{i}-y{m} \end{aligned}$$
(2)

Such approximation can be considered as a truncation of Taylor series expansion of the function Z(u,v) at a point \( (x_{k},y_{m})\).

The calculations of coefficients \(c_{0},c_{1},...,c_{5}\) of local approximation of type (1) are made by methods of the regression analysis according to the coordinate values of the points neighboring P. More precisely, we specify some value of radius r (selected empirically) and consider the set of points for which the following inequality is satisfied (3)

$$\begin{aligned} d_{ij}\le r \end{aligned}$$
(3)

where \(d_{ij}\) is the distance between the point with the coordinates \((x_{i},y_{j},z_{ij})\) and point P, which is defined according to (4)

$$\begin{aligned} d_{ij}=\sqrt{\left( x_{k} -x_{i}\right) ^{2}+\left( y_{m}-y_{j} \right) ^{2}} \end{aligned}$$
(4)

The calculation of the average and Gaussian curvature of the surface is given below. Let r(u,v) parametric surface. The first fundamental form of surface I can be represented as symmetric matrix (5)

$$\begin{aligned} I\left( u,v \right) =\left( u^{T} \right) \cdot \left( \begin{array}{c}E F\\ F G\end{array}\right) \left( \begin{array}{c}v\end{array}\right) \end{aligned}$$
(5)

The first fundamental form completely describes the metric properties of a surface. For example, line element dr can be expressed in terms of the coefficients of the first fundamental form as (6)

$$\begin{aligned} d r^2=E\cdot du^2+2\cdot F\cdot du\cdot dv +G\cdot dv^2 \end{aligned}$$
(6)

The coefficients of the first quadratic form are calculated using the coefficients \(c_{0},c_{1},...,c_{5}\) of the approximating polynomial (1) by formulas (7)–(9)

$$\begin{aligned} E=c_{3}/\sqrt{1+c_{1}^{2}+c_{1}^{2}} \end{aligned}$$
(7)
$$\begin{aligned} F=c_{4}/\sqrt{1+c_{1}^{2}+c_{1}^{2}} \end{aligned}$$
(8)
$$\begin{aligned} G=c_{5}/\sqrt{1+c_{1}^{2}+c_{1}^{2}} \end{aligned}$$
(9)

We will denote the partial derivatives of r with respect to u and v by \( r_{u}\) and \(r_{v}\). Then vector m defined by (10) is a nonzero vector normal to the surface.

$$\begin{aligned} \varvec{m}=\frac{\left[ \varvec{r}_{x},\varvec{r}_{y} \right] }{\left| \left[ \varvec{r}_{x},\varvec{r}_{y} \right] \right| } \end{aligned}$$
(10)

The second fundamental form is a symmetric bilinear form \(\prod \) determined by (11)

$$\begin{aligned} \varvec{r}_{jk}=\frac{\partial ^{2}\varvec{r}}{\partial x_{j}\partial x_{k}} \end{aligned}$$
(11)

The second quadratic form is a symmetric bilinear form \(\prod \) determined by (11)

$$\begin{aligned} \prod \left( u,v) \right) =\left( u^T \right) \cdot \left( \begin{array}{c}LM\\ MN\end{array}\right) \cdot \left( \begin{array}{c}v\end{array}\right) \end{aligned}$$
(12)

Coefficients of second fundamental form, defined by equations (13)–(15) are the functions of coefficients \(c_{1},c_{2}\)

$$\begin{aligned} L=\left( \varvec{r}_{11},\varvec{m} \right) =1+c_{1}^{2} \end{aligned}$$
(13)
$$\begin{aligned} M=\left( \varvec{r}_{12},\varvec{m} \right) =c_{1}\cdot c_{2} \end{aligned}$$
(14)
$$\begin{aligned} N=\left( \varvec{r}_{22},\varvec{m} \right) =1+c_{2}^{2} \end{aligned}$$
(15)

The fundamental role of the second quadratic form in the definition of the local surface geometry is that it allows determining the curvature of the given curve belonging to the surface. Namely, the curvature of the normal section is determined by the formula (16)

$$\begin{aligned} k=\frac{\prod \left( u,v \right) }{I\left( v,v \right) } \end{aligned}$$
(16)

In the tangent plane, we can choose a basis \(\varvec{e}_{1}\varvec{e}_{2}\) in which the forms of I and \(\prod \) at the same time are diagonalized. The directions of vectors \(\varvec{e}_{1}\) and \(\varvec{e}_{2}\) are called principal directions, and they are unequivocally determined if \(k_{1}\ne k_{2}\). The values of \(k_{1}\) and \(k_{2}\) of normal curvatures along the principal directions are called the principal curvatures. They are the extreme values for the normal curvatures at the point, which follows from Euler’s formula (17), where \(\varphi \) is the angle between vectors \(\varvec{e}_{1}\) and \(\varvec{v}\).

$$\begin{aligned} \frac{\prod \left( v,v \right) }{I\left( v,v \right) }=k_{1}\cdot cos^{2}\varphi +k_{2}\cdot sin^{2}\varphi \end{aligned}$$
(17)

The product of the principal curvatures at the point is called the Gaussian curvature of the surface at this point:

$$\begin{aligned} K=k_{1}\cdot k_{2}=\frac{\left( L\cdot N-M^{2} \right) }{\left( E\cdot G-F^{2} \right) } \end{aligned}$$
(18)

The sum of the main curvatures at the point is called the average curvature of the surface at this point:

$$\begin{aligned} H=\frac{1}{2}\cdot \left( k_{1} +k_{2}\right) \end{aligned}$$
(19)

Thus, for each point, the four local curvature values K, H, \(k_{1}\) and \(k_{2}\) can be automatically obtained as the functions of the coefficients of the approximating polynomial. In addition, such curvatures are invariant with respect to the adopted reference system, which represents a very important property in analyzing the surface shape. With simultaneous analysis of the sign and the values of K and H, the classification of the whole point cloud is really achievable, the following surface basic types are possible: hyperbolic \((K = 0,\) parabolic \(((K<0)\), parabolic \((K = 0\), but \(H \ne 0)\), flat \((K = H = 0)\) and elliptic \((K>0)\) [15]. In addition, the method of empirical optimization of the Taylor expansion bandwidth is presented. This makes it possible to calculate the minimum values of K and H, which can be confirmed by an F-criterion, once the error value of the first kind is fixed. After the 3D-models are created by using the point cloud, they are saved for later viewing.

The analysis of publications [16,17,18] related to the development of systems for 3D-scanning based on robot manipulators, and the analysis of solutions presented in the market, it is possible to come to a definite conclusion: the systems of 3D-scanning, where robotic manipulators are applied, use either laser triangulation scanners [18], or optical scanners that are implemented on the basis of machine vision systems (in particular, the system using structured light technology). Currently, a number of manufacturers of optical measuring equipment produce 3D-scanners specially designed for systems using a robot manipulator to move the scanner in space. An analysis of the market suggests that they all use methods of machine vision for scanning, i.e., include multiple cameras, and software allowing for the reconstruction of 3D-scenes, located in the working area of the camera system. As an example, let us note the series of MetraSCAN 3D-optical 3D-scanners produced by Creaform Inc. Machine vision systems require much less time for scanning than point scan systems, that require manipulator passing all the scanning points; in addition, point scanning systems are of little use for scanning objects, the surface of which cannot be described by a smooth function – e.g., polyhedra. On the other hand, the point scanning systems of the same accuracy will always be much cheaper and easier to setup and maintain than the corresponding machine vision system. For any optical scanner, the necessary conditions for achieving high measurement accuracy are a high resolution image sensor and a high quality optical system. Machine vision systems require additional expensive hardware and specific software. At the same time, distance sensors for building a fairly accurate scanning system are cheap and easy to use.

Thus, we propose to generate a robot program by a 3D-model of the object to be processed, obtained in the result of scanning, using an industrial robot as a key component of the scanning system. According to the segmentation results, a set of reference points is selected on the surface; and if we know their spatial coordinates, we will be able to fairly accurately construct the 3D model of the object. After selecting reference points, the software generates the program of the manipulator, accomplishing which the manipulator will successively pass the reference points, performing surface scanning at each of the points.

Thus, this paper summarizes principles of three-dimensional scanning system based on robotic manipulator and laser scanner and describes designing of such device. Solution is universal and does not dependent on actually used devices. This approach brings more flexibility on scanned object; almost any type of object can be scanned. Thus, we are developing an intelligent automated system of controlling an industrial robot manipulator, which allows moving a robot arm along a given 3D-trajectory, a model of the product that the robot will coat with plasma. A distinctive feature of the proposed system is pre-3D-scanning the surface of the rough components or the components being processed.

4 Conclusion

Some specific technical solutions and algorithms for developing an intelligent automated system to control an industrial robot manipulator while plasma cutting and processing of large-size products of complex shapes have been suggested, namely: the algorithms of image segmentation and segmentation of the surface based on the calculation of the local geometric properties of the surface (the method of the segmentation of the point cloud obtained at the stage of rough scanning of the surface) and the multi-view 3D-Reconstruction algorithm to quickly scan an object. We are planning to implement automatic generation of program code of a robot-manipulator, taking into account the data of the 3D-scanning of an object to be processed, previously held by means of distance sensors mounted on the robot manipulator. This will allow using components varying in a wide range of geometric parameters and processing products, whose geometrical parameters are determined with low accuracy or products with deviations from a predetermined shape.