Abstract
As machine learning is applied to increasingly complex tasks, it is likely that the diverse challenges encountered can only be addressed by combining the strengths of different learning algorithms. We examine this aspect of learning through a case study grounded in the robot soccer context. The task we consider is Keepaway, a popular benchmark for multiagent reinforcement learning from the simulation soccer domain. Whereas previous successful results in Keepaway have limited learning to an isolated, infrequent decision that amounts to a turn-taking behavior (passing), we expand the agents’ learning capability to include a much more ubiquitous action (moving without the ball, or getting open), such that at any given time, multiple agents are executing learned behaviors simultaneously. We introduce a policy search method for learning “GetOpen” to complement the temporal difference learning approach employed for learning “Pass”. Empirical results indicate that the learned GetOpen policy matches the best hand-coded policy for this task, and outperforms the best policy found when Pass is learned. We demonstrate that Pass and GetOpen can be learned simultaneously to realize tightly-coupled soccer team behavior.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Benda, M., Jagannathan, V., Dodhiawala, R.: On optimal cooperation of knowledge sources - an empirical investigation. Technical Report BCS–G2010–28, Boeing Advanced Technology Center, Boeing Comp. Serv., Seattle, WA (July 1986)
Chen, M., Foroughi, E., Heintz, F., Huang, Z., Kapetanakis, S., Kostiadis, K., Kummeneje, J., Noda, I., Obst, O., Riley, P., Steffens, T., Wang, Y., Yin, X.: Users manual: RoboCup soccer server — for soccer server version 7.07 and later. The RoboCup Federation (August 2002)
De Boer, P.T., Kroese, D.P., Mannor, S., Rubinstein, R.: A tutorial on the cross-entropy method. Annals of Operations Research 134(1), 19–67 (2005)
Ghavamzadeh, M., Mahadevan, S., Makar, R.: Hierarchical multi-agent reinforcement learning. Aut. Agents and Multi-Agent Sys. 13(2), 197–229 (2006)
Guestrin, C., Lagoudakis, M.G., Parr, R.: Coordinated reinforcement learning. In: Sammut, C., Hoffmann, A.G. (eds.) Proceedings of the Nineteenth International Conference on Machine Learning, University of New South Wales, Sydney, Australia, July 8-12, pp. 227–234. Morgan Kaufmann, San Francisco (2002)
Haykin, S.: Neural Networks: A Comprehensive Foundation. Prentice Hall PTR, Upper Saddle River (1998)
Haynes, T., Wainwright, R., Sen, S., Schoenefeld, D.: Strongly typed genetic programming in evolving cooperation strategies. In: Forrest, S. (ed.) Proc. of the 6th Int. Conf. Gen. Alg., San Mateo,CA, pp. 271–278. Morgan Kaufman, San Francisco (1995)
Iscen, A., Erogul, U.: A new perpective to the keepaway soccer: The takers. In: Proceedings of the Seventh International Joint Conference on Autonomous Agents and Multi–Agent Systems, pp. 1341–1344. International Foundation for Autonomous Agents and Multiagent Systems, Richland (2008)
Jung, T., Polani, D.: Learning Robocup-Keepaway with kernels. In: Gaussian Processes in Practice: JMLR Workshop and Conference Proceedings, vol. 1, pp. 33–57 (2007)
Metzen, J.H., Edgington, M., Kassahun, Y., Kirchner, F.: Analysis of an evolutionary reinforcement learning method in a multiagent domain. In: Proceedings of the Seventh International Joint Conference on Autonomous Agents and Multi–Agent Systems, pp. 291–298. International Foundation for Autonomous Agents and Multiagent Systems, Richland (2008)
Riedmiller, M., Gabel, T.: On experiences in a complex and competitive gaming domain: Reinforcement learning meets robocup. In: 3rd IEEE Symposium on Computational Intelligence and Games, April 2007, pp. 17–23 (2007)
Rosin, C.D., Belew, R.K.: Methods for competitive co-evolution: Finding opponents worth beating. In: Forrest, S. (ed.) Proc. of the 6th Int. Conf. Gen. Alg., pp. 373–380. Morgan Kaufmann, San Mateo (1995)
Stone, P.: Layered Learning in Multiagent Systems: A Winning Approach to Robotic Soccer. MIT Press, Cambridge (2000)
Stone, P., Sutton, R.S., Kuhlmann, G.: Reinforcement learning for RoboCup-soccer keepaway. Adaptive Behavior 13(3), 165–188 (2005)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kalyanakrishnan, S., Stone, P. (2010). Learning Complementary Multiagent Behaviors: A Case Study. In: Baltes, J., Lagoudakis, M.G., Naruse, T., Ghidary, S.S. (eds) RoboCup 2009: Robot Soccer World Cup XIII. RoboCup 2009. Lecture Notes in Computer Science(), vol 5949. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-11876-0_14
Download citation
DOI: https://doi.org/10.1007/978-3-642-11876-0_14
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-11875-3
Online ISBN: 978-3-642-11876-0
eBook Packages: Computer ScienceComputer Science (R0)