Abstract
Since the emergence of deep learning as a dominant technique for numerous tasks in the computer vision domain, the robotics community has strived to utilize its potential. Deep learning represents a framework capable of learning the most complex models necessary to carry out various robotic tasks. We propose to integrate deep learning and one of the fundamental robotic algorithms—visual servoing. Fully convolutional neural networks are used for semantic segmentation, which represents the process of labeling every pixel within the image. The obtained information from labeled (categorical) images can be crucial for mobile robot control in dynamic environments. To adequately utilize semantic segmentation for mobile robot control, the segmented images acquired at the desired and the current pose need to be registered (aligned). Since the accuracy of visual servoing depends on the accuracy of the image registration process, we propose to increase the accuracy of mobile robot positioning by analyzing three different optimization algorithms devoted to the registration of categorical images. The standard gradient descent algorithm is compared to the OnePlusOneEvolutionary algorithm, and simulated annealing. Moreover, different cost functions such as Mattes mutual information, global accuracy, and mean intersection over union are also investigated. All the algorithms are tested on our own wheeled mobile robot RAICO (Robot with Artificial Intelligence based COgnition) developed within the Laboratory for robotics and artificial intelligence. The results indicate that the algorithm with a larger exploration to exploitation ratio provides better results. Moreover, the cost function with the steepest convex domain is more advantageous.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Abbreviations
- RAICO:
-
Robot with artificial intelligence based cognition
- AI:
-
Artificial intelligence
- ML:
-
Machine learning
- DNNs:
-
Deep neural networks
- IBVS:
-
Image-based visual servoing
- PBVS:
-
Position-based visual servoing
- DVS:
-
Direct visual servoing
- CNN:
-
Convolutional neural network
- VGG16:
-
Visual geometry group
- DOF:
-
Degree of freedom
- FCN:
-
Fully convolutional network
- SGD:
-
Stochastic gradient descent
- ReLU:
-
Rectified linear unit
- mIoU:
-
Mean intersection over union
- MI:
-
Mutual information
- GA:
-
Global accuracy
- SA:
-
Simulated annealing
- GD:
-
Gradient descent
- EV:
-
OnePlusOneEvolutionary
References
Sünderhauf N et al (2018) The limits and potentials of deep learning for robotics. Int J Rob Res 37(4–5):405–420. https://doi.org/10.1177/0278364918770733
Mitić M, Vuković N, Petrović M, Miljković Z (2018) Chaotic metaheuristic algorithms for learning and reproduction of robot motion trajectories. Neural Comput Appl 30(4):1065–1083. https://doi.org/10.1007/s00521-016-2717-6
Vuković N, Mitić M, Miljković Z (2015) Trajectory learning and reproduction for differential drive mobile robots based on GMM/HMM and dynamic time warping using learning from demonstration framework. Eng Appl Artif Intell 45:388–404. https://doi.org/10.1016/j.engappai.2015.07.002
Miljković Z, Vuković N, Mitić M, Babić B (2013) New hybrid vision-based control approach for automated guided vehicles. Int J Adv Manuf Technol 66(1–4):231–249. https://doi.org/10.1007/s00170-012-4321-y
Petrović M, Miljković Z, Jokić A (2019) A novel methodology for optimal single mobile robot scheduling using whale optimization algorithm. Appl Soft Comput 81:105520. https://doi.org/10.1016/j.asoc.2019.105520
Petrović M, Miljković Z, Babić B, Vuković N, Čović N (2012) Towards a conceptual design of intelligent material transport using artificial intelligence. Strojarstvo 54(3):205–219
Petrović M, Miljković Z, Babić B (2013) Integration of process planning, scheduling, and mobile robot navigation based on TRIZ and multi-agent methodology. FME Trans 41(2):120–129
Pierson HA, Gashler MS (2017) Deep learning in robotics: a review of recent research. Adv Rob 31(16):821–835. https://doi.org/10.1080/01691864.2017.1365009
Levine S, Pastor P, Krizhevsky A, Ibarz J, Quillen D (2018) Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. Int J Rob Res 37(5):421–436. https://doi.org/10.1177/0278364917710318
Sadeghi F, Levine S (2016) CAD2RL: real single-image flight without a single real image. arXiv Prepr. arXiv:1611.04201
Tobin J, Fong R, Ray A, Schneider J, Zaremba W, Abbeel P (2017) Domain randomization for transferring deep neural networks from simulation to the real world. In: IEEE International conference on intelligent robots and systems (IROS), pp 23–30. https://doi.org/10.1109/IROS.2017.8202133
Chaumette F, Hutchinson S (2006) Visual servo control. I. Basic approaches. IEEE Rob Autom Mag 13(4):82–90. https://doi.org/10.1109/MRA.2006.250573
Jokić A, Petrović M, Miljković Z (2018) Methods for visual servoing of robotic systems: a state of the art survey. Tehnika 73(6):801–816. https://doi.org/10.5937/tehnika1806801j
Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: 3rd International conference on learning representations (ICLR), pp 1–14
Sadeghi F, Toshev A, Jang E, Levine S (2018) Sim2Real viewpoint invariant visual servoing by recurrent control. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 4691–4699. https://doi.org/10.1109/CVPR.2018.00493
Petrović M, Mystkowski A, Jokić A, Đokić L, Miljković Z (2020) Deep learning-based algorithm for mobile robot control in textureless environment. In: 2020 IEEE International conference mechatronic systems and materials (MSM), pp 1–4. https://doi.org/10.1109/msm49833.2020.9201666
Bateux Q, Marchand E, Leitner J, Chaumette F (2018) Training deep neural networks for visual servoing. In: 2018 IEEE International conference on robotics and automation (ICRA), pp 1–8, [Online]. Available: https://hal.inria.fr/hal-01716679/document
Krizhevsky A, Sutskever I, Hinton GE (2020) ImageNet classification with deep convolutional neural networks. In: Advances in neural information processing systems (NIPS), pp 1097–1105. Accessed: 18 Aug 2020 [Online]. Available: http://code.google.com/p/cuda-convnet/
Russakovsky O et al (2015) ImageNet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252. https://doi.org/10.1007/s11263-015-0816-y
Saxena A, Pandya H, Kumar G, Gaud A, Krishna KM (2017) Exploring convolutional networks for end-to-end visual servoing, [Online]. Available: http://arxiv.org/abs/1706.03220
Dosovitskiy A et al. (2015) FlowNet: learning optical flow with convolutional networks. In: IEEE international conference on computer vision, pp 2758–2766
Zhang F, Leitner J, Milford M, Upcroft B, Corke P (2015) Towards vision-based deep reinforcement learning for robotic motion control. arXiv:1511.03791
Zhang F, Leitner J, Ge Z, Milford M, Corke P (2019) Adversarial discriminative sim-to-real transfer of visuo-motor policies. Int J Rob Res 38(10–11):1229–1245
Mitić M, Miljković Z (2014) Neural network learning from demonstration and epipolar geometry for visual control of a nonholonomic mobile robot. Soft Comput 18(5):1011–1025. https://doi.org/10.1007/s00500-013-1121-8
Miljković Z, Mitić M, Lazarević M, Babić B (2013) Neural network reinforcement learning for visual control of robot manipulators. Expert Syst Appl 40(5):1721–1736. https://doi.org/10.1016/j.eswa.2012.09.010
Mithun P, Mehta SA, Shah SV, Bhatnagar G (2020) Student mixture model based visual servoing. arXiv:2006.11347v1
Crombez N, Caron G, Mouaddib EM (2015) Photometric Gaussian mixtures based visual servoing. In: 2015 IEEE/RSJ International conference on intelligent robots and systems (IROS), pp 5486–5491. https://doi.org/10.1109/IROS.2015.7354154
Marchand E (2019) Subspace-based direct visual servoing. IEEE Rob Autom Lett 4(3):2699–2706
Kallem V, Swensen JP, Hager GD, Cowan NJ (2007) Kernel-based visual servoing. In: IEEE/RSJ International conference on intelligent robots and systems, pp 1975–1980
Collewet C, Marchand E (2011) Photometric visual servoing. IEEE Trans Rob 27(4):828–834. https://doi.org/10.1109/TRO.2011.2112593
Bateux Q, Marchand E (2016) Particle filter-based direct visual servoing. In: IEEE International conference on intelligent robots and systems (IROS), pp 4180–4186. https://doi.org/10.1109/IROS.2016.7759615
Dame A, Marchand E (2011) Mutual information-based visual servoing. IEEE Trans Rob 27(5):958–969. https://doi.org/10.1109/TRO.2011.2147090
Bertrand D, Marchand E (2012) Visual servoing using the sum of conditional variance. In: Intelligent robots and systems (IROS), pp 1689–1694
Teulière C, Marchand E (2014) A dense and direct approach to visual servoing using depth maps. IEEE Trans Rob 30(5):1242–1249
Garcia-Garcia A, Orts-Escolano S, Oprea S, Villena-Martinez V, Garcia-Rodriguez J (2017) A review on deep learning techniques applied to semantic segmentation, [Online]. Available: http://arxiv.org/abs/1704.06857
Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 3431–3440. https://doi.org/10.1109/CVPR.2015.7298965
Mitić M, Vuković N, Petrović M, Miljković Z (2015) Chaotic fruit fly optimization algorithm. Knowl Based Syst 89:446–458. https://doi.org/10.1016/j.knosys.2015.08.010
Miljković Z, Petrović M (2017) Application of modified multi-objective particle swarm optimisation algorithm for flexible process planning problem. Int J Comput Integr Manuf 30(2–3):271–291. https://doi.org/10.1080/0951192X.2016.1145804
Petrović M, Mitić M, Vuković N, Miljković Z (2016) Chaotic particle swarm optimization algorithm for flexible process planning. Int J Adv Manuf Technol 85(9–12):2535–2555. https://doi.org/10.1007/s00170-015-7991-4
Petrović M, Vuković N, Mitić M, Miljković Z (2016) Integration of process planning and scheduling using chaotic particle swarm optimization algorithm. Expert Syst Appl 64:569–588. https://doi.org/10.1016/j.eswa.2016.08.019
Styner M, Brechbuhler C, Szekely G, Gerig G (2000) Parametric estimate of intensity inhomogeneities applied to MRI. IEEE Trans Med Imaging 19(3):153–165. https://doi.org/10.1109/42.845174
Acknowledgements
This work has been financially supported by the Ministry of Education, Science and Technological Development of the Serbian Government, through the project “Integrated research in macro, micro, and nano mechanical engineering—Deep learning of intelligent manufacturing systems in production engineering”, under the contract number 451-03-9/2021-14/200105, and by the Science Fund of the Republic of Serbia, Grant No. 6523109, AI - MISSION4.0, 2020–2022.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Miljković, Z., Jokić, A., Petrović, M. (2021). Image Registration Algorithm for Deep Learning-Based Stereo Visual Control of Mobile Robots. In: Koubaa, A., Azar, A.T. (eds) Deep Learning for Unmanned Systems. Studies in Computational Intelligence, vol 984. Springer, Cham. https://doi.org/10.1007/978-3-030-77939-9_13
Download citation
DOI: https://doi.org/10.1007/978-3-030-77939-9_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-77938-2
Online ISBN: 978-3-030-77939-9
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)