Abstract
In service computing, online services and the Internet environment are evolving over time, which poses a challenge to service composition for adaptivity. In addition, high efficiency should be maintained when faced with massive candidate services. Consequently, this paper presents a new model for large-scale and adaptive service composition based on multi-agent reinforcement learning. The model integrates on-policy reinforcement learning and game theory, where the former is to achieve adaptability in a highly dynamic environment with good online performance, and the latter is to enable multiple agents to work for a common task (i.e., composition). In particular, we propose a multi-agent SARSA (State-Action-Reward-State-Action) algorithm which is expected to achieve better performance compared with the single-agent reinforcement learning methods in our composition framework. The features of our approach are demonstrated by an experimental evaluation.
Chapter PDF
Similar content being viewed by others
Keywords
- Reinforcement Learning
- Service Composition
- Reward Function
- Candidate Service
- Reinforcement Learning Method
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Ardagna, D., Pernici, B.: Adaptive service composition in flexible processes. IEEE Transactions on Software Engineering 33(6), 369–384 (2007)
Beauche, S., Poizat, P.: Automated service composition with adaptive planning. In: Bouguettaya, A., Krueger, I., Margaria, T. (eds.) ICSOC 2008. LNCS, vol. 5364, pp. 530–537. Springer, Heidelberg (2008)
Busoniu, L., Babuska, R., De Schutter, B.: A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews 38(2), 156–172 (2008)
Claus, C., Boutilier, C.: The dynamics of reinforcement learning in cooperative multiagent systems. In: AAAI/IAAI, pp. 746–752 (1998)
Gutierrez-Garcia, J.O., Sim, K.-M.: Agent-based service composition in cloud computing. In: Kim, T.-h., Yau, S.S., Gervasi, O., Kang, B.-H., Stoica, A., Ślęzak, D. (eds.) GDC and CA 2010. CCIS, vol. 121, pp. 1–10. Springer, Heidelberg (2010)
Hu, J., Wellman, M.P.: Multiagent reinforcement learning: theoretical framework and an algorithm. In: ICML, vol. 98, pp. 242–250. Citeseer (1998)
Jureta, I.J., Faulkner, S., Achbany, Y., Saerens, M.: Dynamic web service composition within a service-oriented architecture. In: IEEE International Conference on Web Services, ICWS 2007, pp. 304–311. IEEE (2007)
Könönen, V.: Asymmetric multiagent reinforcement learning. Web Intelligence and Agent Systems 2(2), 105–121 (2004)
Littman, M.L.: Markov games as a framework for multi-agent reinforcement learning. In: ICML, vol. 94, pp. 157–163 (1994)
Littman, M.L.: Value-function reinforcement learning in markov games. Cognitive Systems Research 2(1), 55–66 (2001)
Maamar, Z., Mostefaoui, S.K., Yahyaoui, H.: Toward an agent-based and context-oriented approach for web services composition. IEEE Transactions on Knowledge and Data Engineering 17(5), 686–697 (2005)
Monderer, D., Shapley, L.S.: Fictitious play property for games with identical interests. Journal of Economic Theory 68(1), 258 (1996)
Moustafa, A., Zhang, M.: Multi-objective service composition using reinforcement learning. In: Basu, S., Pautasso, C., Zhang, L., Fu, X. (eds.) ICSOC 2013. LNCS, vol. 8274, pp. 298–312. Springer, Heidelberg (2013)
Oh, S.C., Lee, D., Kumara, S.R.: Effective web service composition in diverse and large-scale service networks. IEEE Transactions on Services Computing 1(1), 15–32 (2008)
Panait, L., Luke, S.: Cooperative multi-agent learning: The state of the art. In: Proceedings of 2005 Autonomous Agents and Multi-Agent Systems(AAMAS), vol. 11(3), pp. 387–434 (November 2005)
Papadopoulos, P., Tianfield, H., Moffat, D., Barrie, P.: Decentralized multi-agent service composition. Multiagent and Grid Systems 9(1), 45–100 (2013)
Rummery, G.A., Niranjan, M.: On-line Q-learning using connectionist systems. University of Cambridge, Department of Engineering (1994)
Sutton, R.S., Barto, A.G.: Reinforcement learning: An introduction, vol. 1. Cambridge Univ. Press (1998)
Wang, H., Wang, X.: A novel approach to large-scale services composition. In: Ishikawa, Y., Li, J., Wang, W., Zhang, R., Zhang, W. (eds.) APWeb 2013. LNCS, vol. 7808, pp. 220–227. Springer, Heidelberg (2013)
Wang, H., Zhou, X., Zhou, X., Liu, W., Li, W., Bouguettaya, A.: Adaptive service composition based on reinforcement learning. In: Maglio, P.P., Weske, M., Yang, J., Fantinato, M. (eds.) ICSOC 2010. LNCS, vol. 6470, pp. 92–107. Springer, Heidelberg (2010)
Wang, X., Sandholm, T.: Reinforcement learning to play an optimal nash equilibrium in team markov games. In: NIPS, vol. 15, pp. 1571–1578 (2002)
Xu, W., Cao, J., Zhao, H., Wang, L.: A multi-agent learning model for service composition. In: 2012 IEEE Asia-Pacific Services Computing Conference (APSCC), pp. 70–75. IEEE (2012)
Young, H.P.: The evolution of conventions. Econometrica 61(1), 57–84 (1993)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Wang, H., Chen, X., Wu, Q., Yu, Q., Zheng, Z., Bouguettaya, A. (2014). Integrating On-policy Reinforcement Learning with Multi-agent Techniques for Adaptive Service Composition. In: Franch, X., Ghose, A.K., Lewis, G.A., Bhiri, S. (eds) Service-Oriented Computing. ICSOC 2014. Lecture Notes in Computer Science, vol 8831. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-45391-9_11
Download citation
DOI: https://doi.org/10.1007/978-3-662-45391-9_11
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-662-45390-2
Online ISBN: 978-3-662-45391-9
eBook Packages: Computer ScienceComputer Science (R0)