Abstract
In this paper, hybrid integrated dynamic control algorithm for humanoid locomotion mechanism is presented. The proposed structure of controller involves two feedback loops: model-based dynamic controller including impart-force controller and reinforcement learning feedback controller around zero-moment point. The proposed new reinforcement learning algorithm is based on modified version of actor-critic architecture for dynamic reactive compensation. Simulation experiments were carried out in order to validate the proposed control approach.The obtained simulation results served as the basis for a critical evaluation of the controller performance.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Atkeson, C.G., Santamaria, J.C.: A comparison of direct and model-based reinforcement learning. In: Proceedings of the 1997 IEEE International Conference on Robotics and Automation, pp. 3557–3564. Albuquerque, USA (1997)
Benbrahim, H., Franklin, J.A.: Biped dynamic walking using reinforcement learning. Robot. Auton. Syst. 22, 283–302 (1997)
Berenji, H.R., Khedkar, P.: Learning and tuning fuzzy logic controllers through reinforcements. IEEE Trans. Neural Netw. 3, 724–740 (1992)
Bertsekas, D.P., Tsitsiklis, J.N.: Neuro-Dynamic Programming. Athena Scientific, Belmont, USA (1996)
Chew, C., Pratt, G.A.: Dynamic bipedal walking assisted by learning. Robotica 20, 477–491 (2002)
Doya, K.: Reinforcement learning in continuous time and space. Neural Comput. 12, 219–245 (2000)
Gienger, M., Löffler, K., Pfeiffer, F.: Walking control of a biped robot based on inertial measurement. In: Proceedings of IARP International Workshop on Humanoid and Human Friendly Roboticsm, pp. 22–29. Tsukuba, Japan (2002)
Gullapalli, V.: A stochastic reinforcement learning algorithm for learning real-valued functions. Neural Netw. 3, 671–692 (1990)
Gullapalli, V., Franklin, J.A., Benbrahim, H.: Acquiring robot skills via reinforcement learning. IEEE Control Systems Magazine, pp. 13–24 (1994)
Hirai, K., Hirose, M., Haikawa, Y., Takenaka, T.: The development of honda humanoid robot. In: Proceedings of the 1998 IEEE Int. Conference on Robotics and Automation, pp. 1321–1326 (1998)
Kamio, S., Iba, H.: Adaptation technique for integrating genetic programming and reinforcement learning for real robot. IEEE Trans. Evol. Comput. 9(3), 318–333 (June 2005)
Katić, D., Vukobratović, M.: Survay of intelligent control techniques for humanoid robots. J. Intell. Robot. Syst. 37, 117–141 (2003)
Katić, D., Vukobratović, M.: Intelligent Control of Robotic Systems. Kluwer, Dordrecht, The Netherlands (2003)
Katic, D., Vukobratovic, M.: Survey of intelligent control algorithms for humanoid robots. In: Proceedings of the 16th IFAC World Congress, Prague, Czech Republic, July 2005
Li, Q., Takanishi, A., Kato, I.: Learning control of compensative trunk motion for biped walking robot based on ZMP. In: Proceedings of the 1992 IEEE/RSJ Intl. Conference on Intelligent Robot and Systems, pp. 597–603 (1992)
Mori, T., Nakamura, Y., Sato, M., Ishii, S.: Reinforcement learning for a cpg-driven biped robot. In: Proceedings of the Nineteenth National Conference on Artificial Intelligence (AAAI), pp. 623–630 (2004)
Morimoto, J., Cheng, G., Atkeson, C.G., Zeglin,G.: A simple reinforcement learning algorithm for biped walking. In: Proceedings of the 2004 IEEE International Conference on Robotics & Automation. New Orleans, USA (2004)
Nakamura, Y., Sato, M., Ishii, S.: Reinforcement learning for biped robot. In: Proceedings of the 2nd International Symposium on Adaptive Motion of Animals and Machines (2003)
Park H.J., Chung H.: Hybrid control for biped robots using impedance control and computed-torque control. In: Proceedings of the 1999 IEEE International Conference on Robotics and Automation, pp. 1365–1370. Detroit, USA (1999)
Park, H.J.: Impedance control for biped robot locomotion. IEEE Trans. Robot. Autom. 17(6), 10–6 (2001)
Peters, J., Vijayakumar, S., Schaal, S.: Reinforcement learning for humanoid robots. In: Proceedings of the Third IEEE-RAS International Conference on Humanoid Robots, Karlsruhe & Munich (2003)
Salatian, A.W., Yi, K.Y., Zheng, Y.F.: Reinforcement learning for a biped robot to climb sloping surfaces. J. Robot. Syst. 14, 283–296 (1997)
Sugihara, T., Nakamura, Y.: Whole-body cooperative balancing of humanoid robot using COG Jacobian. In: Proceedings of the 2002 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2575–2580, Lausanne (2002)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. The MIT Press, Cambridge, USA (1998)
Tedrake, R., Zhang, T.W., Seung, H.S.: Stochastic policy gradient reinforcement learning on a simple 3d biped. In: Proceedings of the 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (2004)
Vukobratović, M., Borovac, B., Surla, D., Stokić, D.: Biped Locomotion - Dynamics, Stability, Control and Application. Springer, Berlin, Germany (1990)
Vukobratovic, M., Ekalo, Y.: New approach to control of robotic manipulators interacting with dynamic environment. Robotica 14, 31–39 (1996)
Vukobratović, M., Borovac, B.: Zero-moment point – thirty five years of its life. Int. J. Humanoid Robot 1, 157–173 (2004)
Vukobratović, M., Borovac, B.: Note on the article “zero-moment point – thirty five years of its life”. Int. J. Humanoid Robot. 2, 225–227 (2005)
Watkins, C.J.C.H., Dayan, P.: Q learning. Mach. Learn. 8, 279–292 (1992)
Yokoi, F., Kanehiro, F., Kaneko, K., Fujiwara, K., Kajita, S., Hirukawa, H.: Experimental study of biped locomotion of humanoid robot HRP-1S. In: Siciliano, B., Dario, P. (eds.) Experimental Robotics VIII, pp. 75–84. Springer, Berlin, Germany (2003)
Zhou, C., Meng, Q.: Reinforcement learning and fuzzy evaluative feedback for a biped robot. In: Proceedings of the 2000 IEEE International Conference on Robotics and Automation, pp. 3829–3834. San Francisco, USA (2000)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Katić, D.M., Rodić, A.D. & Vukobratović, M.K. Hybrid Dynamic Control Algorithm for Humanoid Robots Based on Reinforcement Learning. J Intell Robot Syst 51, 3–30 (2008). https://doi.org/10.1007/s10846-007-9174-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10846-007-9174-5