Abstract
In the paper, the vision system for control a electro-hydraulic manipulator is presented. The authors have proposed a Kalman Filter (KF) to estimate a central position of markers. Two different methods of initial estimation of markers position are used. First one is based on central point of marker’s mass. Second refers to a circle fitting method of binary object. These two methods theoretically give the same position of a central point of given marker, however, in case of image distortion results will have different errors. This can occur when operator is holding marker in a hand, and cover some parts of it. Therefore it is important to develop new robust method for marker tracking. Authors proposed Kalman Filter to estimate central point of a marker by making a fusion of information provided by two initial estimation methods. The conducted research proved that KF reduces total manipulator’s control error even in situation where \(30\,\%\) of marker area is invisible.
Please note that the LNCS Editorial assumes that all authors have used the western naming convention, with given names preceding surnames. This determines the structure of the names in the running heads and the author index.
Access provided by Autonomous University of Puebla. Download conference paper PDF
Similar content being viewed by others
Keywords
- Vision system
- Camera
- Geometric fit
- Pratt
- Kasa
- Taubin
- Kalman filter
- Marker detection
- Electro-hydraulic manipulator
1 Introduction
Nowadays vision systems are fest-growing technology which is getting more and more usefull in everyday of life (mobile phones—camera with face recognition algorithm, smart TV—geasture control as well as industry e.g. quality control [1], products classification [2], monitoring systems [3], items sorting in cooperation with robots [4, 5]. Many of them need a special algorithms which will be robust for disturbances. These algorithms are divided into two groups. First refers to features detection [6, 7]. Second group is focused on marker detection. Also three groups of markers are known. The passive markers [8], active [9, 10] and coded [11]. Passive markers are made of color paper or special material which allows to reflect the light, it is usually used in 3D measurement vision system [12]. The reflective marker is illuminated by infrared light, which should be robust for Sun rays. In many cases, when some parts of markers (either passive or active) are invisible, the recognition is difficult, or in worst scenerio impossible. Therefore most popular markers have circle-like shape [13], it is claimed that this is an optimal configuration [14]. However, when the user accidentally covers small parts of it, the coupled vision system does not guarantee that markers will be well captured. That can yield problems with detection and, what is more, tracking of markers central points. Thus it is important to search for a new implementable algorithm to recognize and filter markers features. The presented study is a continuation of a research given in [15]. The authors proved that in the problem of marker’s position tracking, the Kalman Filter turned out to be more accurate than complementary filter. Therefore here, only the previously undertaken methodology will be considered and applied in control of a electro-hydraulic manipulator.
2 The Testbed
The tests were performed on a electro-hydraulic manipulator equipped in proportional valves, an industrial controller and the vision system.
Images were captured by a fast camera 3iCube IC1500CU (Fig. 1) [16] with special software called Adaptive Vision Studio, which uses SEE multicore instructions [17]. This camera was connected to the PC computer by USB 3.0. The focal length of the camera lens is 3.5 mm. The max resolution of camera is equal to \(2592 \times 1944\) yielding 15 fps, but in current research the camera’s resolution was reduced to \(640\times 480\) px and image capture was performed with frequency up to 50 Hz. Calculations were performed on the personal computer. Next, the data with coordinates were sent to PLC (Programmable Logic Controller—B&R company) via TCP/IP protocol. The position of hydraulic cylinder was controlled by proportional valves connected with dedicated control card. Input signal in the card was in a range of \(-\)10..\(+\)10 VDC. The positions of angles of manipulator’s arms were measured by incremental encoders with resolution of 3600 impulses per revolution. The block diagram of the stand is shown in Fig. 2.
The main algorithms of image processing and estimation of a central position of the markers is computed in two phases. The first phase is a basic processing of image:
-
image capture from the camera (RGB image),
-
Gaussian smoothing,
-
HSV filtration,
-
image thresholding to region,
-
region erode,
-
region dilate,
-
split region into blobs.
The second phase refers to marker’s features extraction. In this phase, a few steps can be highlighted. Firstly it is necessary to calculate the size of region:
where: p—is a position of the point on an image, \(n \times m\)—is a size of region, \(p(i,j)=1\)—when object exist, \(p(i,j) = 0\)—when object does not exist, Next, the center of mass of the region (\(x_m,y_m\)) must be found. It can be simply calculated with the following:
In the literature several fitting algorithms can be found. All of them find best circle and based on that, the central points of the markers are established. The most popular are proposed by Kasa [18], Pratt [19] and Taubin [20]. The main task of those algorithms is to minimize the cost of function:
where: a, b—center of circle, R—radius of circle.
The performance of these algorithms was verified by Al-Sharadhad in [21]., but Chernov proved, that algorithms proposed by the Pratt and Taubin are more stable and accurate than the Kasa [22]. Therefore in this article the Authors will focus only on Pratt’s algorithm.
3 Implementation of Kalman Filter
The Kalman Filter is well-known recursive algorithm, which was originally used for linear problems [23]. During the last fifty years, the KF evaluated to deal with nonlinear models, however here the classic type of the KF is used. The overall framework of the KF assumes two-stages process. In the first one, namely prediction, the model data is exploited. During that stage some forecasts on state vector behavior are made. In next stage, the corrections are being imposed. The KF operate on process model which can be written in a state-space representation:
where \(\underline{x}_k\) denotes state vector, A, B and C refer to process, input and output matrices, respectively. In the fusion process proposed by the authors, the state vector contains positions (x,y) and velocities (\(v_x\),\(v_y\)) of a central of the marker.
Matrices A and B are time-variant and thus they were extended with subscripts. The main process is incorporated in \(A_k\) and \(B_k\). It simply reflects the dynamics of the central point and it is given by:
where: \(\mathbf {A}_k^z = \left[ \begin{array}{c c} 1 &{} dt \\ - \frac{1}{dt} &{} 0\\ \end{array} \right] ,\) \(\mathbf {B}_k^z = \left[ \begin{array}{c } 0 \\ \frac{1}{dt}\\ \end{array} \right] ,\)
The output matrix \(C_k\) directly transfers the state vector to the output vector:
The Kalman Filter equations are divided into two stages: The first is state prediction, which yields the a priori state vector \(\underline{\hat{x}}_k^-\), next it is necessary to compute a priori covariance matrix \(P_k^-\). One will notice, that by applying the first estimation of central point positions in \(\underline{u}_k\) the a priori estimation of velocities will be calculated. The KF is prepared in that way, so it can estimate the velocity of point coordinates (in X and Y axis) based on previous (k-1) a posteriori positions and the input. Also the positions are upgraded by integrating the a posteriori velocities.
In the second stage of Kalman Filter, namely correction, three main tasks are being executed. The first one is the Kalman gain \(K_k\) calculation, then the prior state vector \(\underline{\hat{x}}_k\) is corrected with measurement innovation. Finally the covariance matrix \(P_k\) is updated, obtaining the a posteriori covariance \(P_k\).
The Kalman Filter is tunned via the \(Q_k\) and \(R_k\) matrices. The \(Q_k\) refers to process noise, while the \(R_k\) stand for measurement noise.
4 The Experiment
The main aim of the research was to verify a new method of markers position estimation in case of simulated disturbances like covering or shading. During the experiment the saved video was used with dynamic disturbance like covering some parts of marker.
Similar situation can occur when a person holds the marker in a hand and makes some movements. The incomplete information of marker’s position can be observed. The offline study with saved video is useful in comparison of obtained results via different methods. The recorded track for the same for all experiments. In order to capture an image in a wide area, the camera was placed in a distance of 1.3 m against the marker. In the Fig. 3 the processed image and manipulator’s trajectory were shown. As it was presented in the Fig. 3a, three red markers were used to control an electro-hydraulic manipulator. It simply increases efficient, while only one color must be filtrated. In the bottom right hand corner two markers are visible. Those markers are used to scale pixels from the image to manipulator’s metric units. The last marker is directly used to control the tool center point (TCP) of the manipulator.
The image in processing application was shown in Fig. 4. A background of image was deleted to extract marker’s features. Three points placed around markers were used to circle fitting algorithm with results in the fitted circles. In the center of markers a results of a center of mass algorithm was presented. In Fig. 5 the output charts from image processing and output of control of the electro-hydraulic manipulator were shown. As it can be noted, in Fig. 3b a fluctuating pixel in X axis in time domain in occurance of disturbances has appeared. In Fig. 5b an angle computed from inverse kinematic was presented. The last, Fig. 6 is the output trajectory from electro-hydraulic manipulator.
In Fig. 7 absolute positions were compared, these remains the most interesting result. In Fig. 7a the position computed from vision system were presented. First positions were taken from pixels, next inverse kinematic was computed. To check results, forwards kinematic was computed. The fluctuating noise on chart is visible, because the reference signal with scale was taken from two markers which were disrupted by position measurement errors.
The output trajectory from the manipulator was shown in Fig. 7b. The line is more stable, because electro-hydraulic drives are in general inertial objects and cause natural signal filtration.
The overall results were presented in Table 1. As it can be seen, the best results are provide by Kalman filter marked by blue color. A comparison of total amplitude of oscillation in both axes i.e. absolute error was shown in Table 1.
The total error of positioning of the TCP in electro-hydraulic manipulator was reduced by more than \(40\,\%\).
5 Conclusions
In the article three different methods i.e. circle fitting (proposed by Pratt), central point of mass, and the Kalman filter applied in estimation of a central position of the marker were presented. The first part of the article recalls state of the art. In the second section the testbed and the methods for markers central point estimation were described. To undertake the experimental studies, the research stand was built. The images were captured from camera with frequency up to 50 Hz and the resolution of \(640\,\times \,480\). In the third section the mathematical equations which were used to implementation of the KF were described. In the fourth section the results of the experiment were shown. First part consist of image processing results and electro-hydraulic manipulator trajectory, therefore the charts with angle and the positions of manipulator tip were compared. The main aim of the research was to implement and compare three estimation methods i.e. circle fitting, center of mass and the Kalman filter. All of these methods returns central points of markers. The KF uses two inputs of estimated positions in X and Y axis from other proposed methods. The results prove that KF method is suitable in case of estimation of markers positions and yields better results than CF or CM. However, when input parameters are over the range, the estimation results cannot be valid and the total error can increase. Because errors can appear caused by incompleteness of the markers positions information, all filters are important. The KF method improves the estimation provided by CF or CM method, also by using the KF the output signal has smooth characteristic. What is more, the solution given in the paper ensures good results even though when some image parts are incomplete or require prior reconstruction. The total error in the worst scenario, (\(30\,\%\) of marker is invisible) was reduced by more than 40 %.
References
Nandi, C.S., Tudu, B., Koley, C.: An automated machine vision based system for fruit sorting and grading. In: 2012 Sixth International Conference on Sensing Technology (ICST), pp. 195–200 (2012)
Akbar, H., Prabuwono, A.S.: The design and development of automated visual inspection system for press part sorting. In: International Conference on Computer Science and Information Technology. ICCSIT 08, pp. 683–686 (2008)
Govardhan, P., Pati, U.C.: NIR image based pedestrian detection in night vision with cascade classification and validation. In: International Conference on Advanced Communication Control and Computing Technologies (ICACCCT), pp. 1435–1438 (2014)
Bodhale, D., Afzulpurkar, N., Thanh, N.T.: Path planning for a mobile robot in a dynamic environment. In: IEEE International Conference on Robotics and Biomimetics (ROBIO), Bangkok,pp. 2115–2120 (2009)
Hong, S.M., Jang, W.S., Son, J.K., Kim, K.S.: Evaluation of two robot vision control algorithms developed based on N-R and EKF methods for the rigid-body placement. In: International Conference on Advanced Intelligent Mechatronics, Wollongong, Australia, July 9–12, pp. 938–943 (2013)
Schomerus, V., Rosebrock, D., Wahl, F.M.: Camera-based lane border detection in arbitrarily structured environments. In: 2014 IEEE Intelligent Vehicles Symposium Proceedings, pp. 56–63 (2014)
Chugo, D., Hirose, K., Nakashima, K., Yokota, S., Kobayashi, H., Hashimoto, H.: Camera-based navigation for service robots using pictographs on the crossing point. In: IECON 2012—38th Annual Conference on IEEE Industrial Electronics Society, pp. 4154–4159 (2012)
Sampe, I.E., Amar Vijai, N., Tati Latifah, R.M., Apriantono, T.: A study on the effects of lightning and marker color variation to marker detection and tracking accuracy in gait analysis system. In: 2009 International Conference on Instrumentation, Communications, Information Technology, and Biomedical Engineering (ICICI-BME), pp. 1–5 (2009)
Kim, D., Choi, J., Park, M.: Detection of multi-active markers and pose for formation control. In: 2010 International Conference on Control Automation and Systems (ICCAS), pp. 943–946 (2010)
Fukuzawa, M., Hama, H., Nakamori, N., Yamada, M.: High-speed distance measurement between moving vehicles with NIR-LED markers. In: 11th International Conference on Computer and Information Technology. ICCIT 2008, pp. 516–520 (2008)
Gherghina, A., Olteanu, A., Tapus, N.: A marker-based augmented reality system for mobile devices. In: Roedunet International Conference (RoEduNet), 2013 11th, pp. 1–6 (2013)
PONTOS—Dynamic 3D Analysis. http://www.gom.com/metrology-systems/system-overview/pontos.html
Mochizuki, Y., Imiya, A., Torii, A.: Circle-marker detection method for omnidirectional images and its application to robot positioning. In: IEEE 11th International Conference on Computer Vision. ICCV 2007, pp. 1–8 (2007)
Khler, J., Pagani, A., Stricker, D.: Detection and Identification Techniques for Markers Used in Computer Vision, in Modeling and Engineering. Bodega Bay, CA, USA, o.A. (2011)
Owczarek, P., Goski, J.: An estimation of central points of circle markers in a vision system by using Kalman filter and complementary filter. In: Proceedings of The 20th International Conference on Methods and Models in Automation and Robotics, between 24th and 27th Aug 2015
NET New Electronic Technology GMBH. http://net-gmbh.com/
Adaptive Vision Studio. https://www.adaptive-vision.com/en/home/’
Corral, C.A., Lindquist, C.S.: On implementing Kasas circle fit procedure. IEEE Trans. Instrum. Meas. 47(3), 789–795 (1998)
Pratt, V.: Direct least-squares fitting of algebraic surfaces. Comput. Graph. 21, 145–152 (1987)
Taubin, G.: Estimation of planar curves, surfaces, and nonplanar space curves defined by implicit equations with applications to edge and range image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 13(11), 1115–1138 (1991)
Ali, N.C.: Al-Sharadqah. Error analysis for circle fitting algorithms. Electron. J. Stat. 3 (2009)
Chernov, N., Lesort, C.: Least squares fitting of circles. J. Math. Imaging Vis. 23(3), 239–252 (2005)
Kalman, R.E.: A new approach to linear filtering and prediction problems. Trans. ASME J. Basic Eng., Ser. D 82, 35–45 (1960)
Acknowledgments
The work is partially sponsored by the project “Scholarship support for PH.D. students specializing in majors strategic for Wielkopolska’s development”, Sub-measure 8.2.2 Human Capital Operational Programme, co-financed by European Union under the European Social Fund.
The work described in this paper was funded from 02/23/DS-PB/1208 project (Nowe techniki w urzadzeniach mechatronicznych).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Owczarek, P., Gośliński, J., Rybarczyk, D., Kubacki, A. (2016). Control of an Electro-Hydraulic Manipulator by Vision System Using Central Point of a Marker Estimated via Kalman Filter. In: Szewczyk, R., Zieliński, C., Kaliczyńska, M. (eds) Challenges in Automation, Robotics and Measurement Techniques. ICA 2016. Advances in Intelligent Systems and Computing, vol 440. Springer, Cham. https://doi.org/10.1007/978-3-319-29357-8_51
Download citation
DOI: https://doi.org/10.1007/978-3-319-29357-8_51
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-29356-1
Online ISBN: 978-3-319-29357-8
eBook Packages: EngineeringEngineering (R0)