Abstract
Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs) provide powerful modeling tools for multiagent decision-making in the face of uncertainty, but solving these models comes at a very high computational cost. Two avenues for side-stepping the computational burden can be identified: structured interactions between agents and intra-agent communication. In this paper, we focus on the interplay between these concepts, namely how sparse interactions impact the communication needs. A key insight is that in domains with local interactions the amount of communication necessary for successful joint behavior can be heavily reduced, due to the limited influence between agents. We exploit this insight by deriving local POMDP models that optimize each agent’s communication behavior. Our experimental results show that our approach successfully exploits sparse interactions: we can effectively identify the situations in which it is beneficial to communicate, as well as trade off the cost of communication with overall task performance.
This work was funded in part by Fundação para a Ciência e a Tecnologia (INESC-ID multiannual funding) through the PIDDAC Program funds and the project CMU-PT/SIA/0023/2009 under the Carnegie Mellon-Portugal Program. M.S. is funded by the FP7 Marie Curie Actions Individual Fellowship #275217 (FP7-PEOPLE-2010-IEF).
Access provided by Autonomous University of Puebla. Download to read the full chapter text
Chapter PDF
Similar content being viewed by others
Keywords
- Multiagent System
- Markov Decision Process
- Partially Observable Markov Decision Process
- Partial Observability
- Primitive Action
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Allen, M., Zilberstein, S.: Agent influence as a predictor of difficulty for decentralized problem-solving. In: Proc. 22nd AAAI Conf. Artificial Intelligence, pp. 688–693 (2007)
Becker, R., Zilberstein, S., Lesser, V., Goldman, C.: Transition-independent decentralized Markov decision processes. In: Proc. Int. Conf. Auton. Agents and Multiagent Systems, pp. 41–48 (2003)
Becker, R., Lesser, V., Zilberstein, S.: Decentralized Markov decision processes with event-driven interactions. In: Proc. Int. Conf. Auton. Agents and Multiagent Systems, pp. 302–309 (2004)
Oliehoek, F., Spaan, M., Whiteson, S., Vlassis, N.: Exploiting locality of interaction in factored Dec-POMDPs. In: Proc. Int. Conf. Auton. Agents and Multiagent Systems (2008)
Spaan, M., Melo, F.: Interaction-driven Markov games for decentralized multiagent planning under uncertainty. In: Proc. Int. Conf. Auton. Agents and Multiagent Systems, pp. 525–532 (2008)
Witwicki, S., Durfee, E.: Influence-based policy abstraction for weakly-coupled Dec-POMDPs. In: Int. Conf. Automated Planning and Scheduling (2010)
Varakantham, P., Kwak, J., Taylor, M., Marecki, J., Scerri, P., Tambe, M.: Exploiting coordination locales in distributed POMDPs via social model shaping. In: Proc. 19th Int. Conf. Automated Planning and Scheduling, pp. 313–320 (2009)
Goldman, C., Zilberstein, S.: Optimizing information exchange in cooperative multiagent systems. In: Proc. 2nd Int. Conf. Autonomous Agents and Multiagent Systems, pp. 137–144 (2003)
Goldman, C., Zilberstein, S.: Communication-based decomposition mechanisms for decentralized MDPs. J. Artificial Intelligence Res. 32, 169–202 (2008)
Roth, M., Simmons, R., Veloso, M.: Decentralized communication strategies for coordinated multiagent policies. In: Multi-Robot Systems: From Swarms to Intelligent Automata, pp. 93–106 (2005)
Roth, M., Simmons, R., Veloso, M.: Exploiting factored representations for decentralized execution in multiagent teams. In: Proc. Int. Conf. Auton. Agents and Multiagent Systems, pp. 469–475 (2007)
Spaan, M., Gordon, G., Vlassis, N.: Decentralized planning under uncertainty for teams of communicating agents. In: Proc. Int. Conf. Auton. Agents and Multiagent Systems (2006)
Tasaki, M., Yabu, Y., Iwanari, Y., Yokoo, M., Tambe, M., Marecki, J., Varakantham, P.: Introducing communication in Dis-POMDPs with locality of interaction. In: IEEE/WIC/ACM Int. Conf. Web Intelligence and Intelligent Agent Technology, vol. 2, pp. 169–175 (2008)
Wu, F., Zilberstein, S., Chen, X.: Multi-agent online planning with communication. In: Proc. Int. Conf. Automated Planning and Scheduling, pp. 321–329 (2009)
Xuan, P., Lesser, V., Zilberstein, S.: Communication decisions in multiagent cooperation: Model and experiments. In: Proc. 5th Int. Conf. Autonomous Agents, pp. 616–623 (2001)
Mostafa, H., Lesser, V.: Offline planning for communication by exploiting structured interactions in decentralized MDPs. In: IEEE/WIC/ACM Int. Conf. Web Intelligence and Intelligent Agent Technology, pp. 193–200 (2009)
Melo, F., Veloso, M.: Decentralized MDPs with sparse interactions. Artificial Intelligence 175(11), 1757–1789 (2011)
Pynadath, D., Tambe, M.: The communicative multiagent team decision problem: Analyzing teamwork theories and models. J. Artificial Intelligence Res. 16, 389–423 (2002)
Becker, R., Carlin, A., Lesser, V., Zilberstein, S.: Analyzing myopic approaches for multi-agent communications. Computational Intelligence 25(1), 31–50 (2009)
Seuken, S., Zilberstein, S.: Formal models and algorithms for decentralized decision making under uncertainty. Auton. Agents and Multi-Agent Systems (2008)
Nair, R., Tambe, M., Yokoo, M., Pynadath, D., Marsella, S.: Taming decentralized POMDPs: Towards efficient policy computation for multiagent settings. In: Proc. 18th Int. Joint Conf. Artificial Intelligence, pp. 705–711 (2003)
Doshi, P., Gmytrasiewicz, P.: On the difficulty of achieving equilibrium in interactive POMDPs. In: Proc. 21st AAAI Conf. Artificial Intelligence, pp. 1131–1136 (2006)
Spaan, M.T.J., Vlassis, N.: Perseus: Randomized point-based value iteration for POMDPs. J. Artificial Intelligence Res. 24, 195–220 (2005)
Oliehoek, F., Spaan, M., Vlassis, N.: Optimal and approximate Q-value functions for decentralized POMDPs. J. Artificial Intelligence Res. 32, 289–353 (2008)
Spaan, M., Oliehoek, F., Amato, C.: Scaling up optimal heuristic search in Dec-POMDPs via incremental expansion. In: Proc. Int. Joint Conf. Artificial Intelligence, pp. 2027–2032 (2011)
Becker, R., Zilberstein, S., Lesser, V., Goldman, C.: Solving transition independent decentralized Markov decision processes. J. Artificial Intelligence Res. 22, 423–455 (2004)
Mostafa, H., Lesser, V.: A compact mathematical formulation for problems with structured agent interactions. In: Proc. AAMAS MSDM Workshop (2011)
Goldmann, C., Allen, M., Zilberstein, S.: Learning to communicate in a decentralized environment. J. Auton. Agents and Multiagent Systems 15(1), 47–90 (2007)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Melo, F.S., Spaan, M.T.J., Witwicki, S.J. (2012). QueryPOMDP: POMDP-Based Communication in Multiagent Systems. In: Cossentino, M., Kaisers, M., Tuyls, K., Weiss, G. (eds) Multi-Agent Systems. EUMAS 2011. Lecture Notes in Computer Science(), vol 7541. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-34799-3_13
Download citation
DOI: https://doi.org/10.1007/978-3-642-34799-3_13
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-34798-6
Online ISBN: 978-3-642-34799-3
eBook Packages: Computer ScienceComputer Science (R0)