Abstract
Deep reinforcement learning is an active research point in the academic community now more than ever. The computational capabilities and the deep neural networks revolution can automatically find compact low-dimensional features of high-dimensional data. As the convolutional neural network gets better at dealing with the visual world, the more outstanding results show reinforcement-learning agents that can concur with various problems. Intending to develop stronger agents and go beyond the benchmark, we need environments that provide more challenge than the problems that already have solved. Real-time strategy games considered as perfect challenging environment for reinforcement learning agents because of their vast state-action spaces that mimic the state and action spaces in real-world problems. Besides, real-time strategy games are simulations with complicated interactions, so they have no cost when applying trial and error learning techniques. Deep reinforcement learning agents considered a promising step towards fully autonomous agents that learn from trial and error with little or no prior knowledge about the environment they were dealing with. The goal of this survey is to state the current state of the art of deep reinforcement learning (DRL) research in the real-time strategy game “StarCraft” introducing the creative ways in which neural networks can be used to bring us steps closer to create AI agents that can deal with the real world. Our survey will cover a theoretical background of reinforcement learning and discuss the families of reinforcement learning algorithms. We will also explain the deep reinforcement learning framework and its contributions to StarCraft. To conclude, we ask the following research question: How can deep reinforcement learning algorithms play StarCraft's full game?
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
gym.openai.com.
References
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017). A brief survey of deep reinforcement learning. arXiv preprint arXiv:1708.05866
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017). Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6), 26–38.
Bertsekas, D. P. (2005). Dynamic programming and suboptimal control: A survey from ADP to MPC. European Journal of Control, 11(4–5), 310–334.
Boutilier, C., & Brafman, R. I. (1997, July). Planning with concurrent interacting actions. In AAAI/IAAI (pp. 720–726).
Brown, N., Sandholm, T., & Machine, S. (2017, August). Libratus: The Superhuman AI for No-Limit Poker. In IJCAI (pp. 5226–5228).
Butler, S., & Demiris, Y. (2010, August). Partial observability during predictions of the opponent's movements in an RTS game. In Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games (pp. 46–53). IEEE.
Čertický, M., & Churchill, D. (2017, September). The current state of StarCraft AI competitions and bots. In Thirteenth Artificial Intelligence and Interactive Digital Entertainment Conference.
Churchill, D. G. (2016). Heuristic search techniques for real-time strategy games.
Corder, K., Vindiola, M. M., & Decker, K. (2019). Decentralized multi-agent actor-critic with generative inference. arXiv preprint arXiv:1910.03058.
Da Silva, F. L., Warnell, G., Costa, A. H. R., & Stone, P. (2020). Agents teaching agents: A survey on inter-agent transfer learning. Autonomous Agents and Multi-Agent Systems, 34(1), 9.
Dahl, G. E., Yu, D., Deng, L., & Acero, A. (2011). Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. IEEE Transactions on Audio, Speech, and Language Processing, 20(1), 30–42.
Denton, E., Gross, S., & Fergus, R. Semi-supervised learning with context-conditional generative adversarial networks. arXiv preprint arXiv:1611.06430.
Diuk, C., Cohen, A., & Littman, M. L. (2008, July). An object-oriented representation for efficient reinforcement learning. In Proceedings of the 25th International Conference on Machine learning (pp. 240–247). ACM.
Foerster, J. N., Farquhar, G., Afouras, T., Nardelli, N., & Whiteson, S. (2018, April). Counterfactual multi-agent policy gradients. In Thirty-Second AAAI Conference on Artificial Intelligence.
Grondman, I., Busoniu, L., Lopes, G. A., & Babuska, R. (2012). A survey of actor-critic reinforcement learning: Standard and natural policy gradients. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 42(6), 1291–1307
Gu, Z., Jia, Z., & Choset, H. (2019) Adversary a3c for robust reinforcement learning. arXiv preprint arXiv:1912.00330.
Horgan, D., Quan, J., Budden, D., Barth-Maron, G., Hessel, M., Van Hasselt, H., & Silver, D. (2018). Distributed prioritized experience replay. arXiv preprint arXiv:1803.00933.
Hou, Y., Liu, L., Wei, Q., Xu, X., & Chen, C. (2017, October). A novel DDPG method with prioritized experience replay. In 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (pp. 316–321). IEEE (2017).
Kim, M. J., Kim, K. J., Kim, S., & Dey, A. K. (2018). Performance evaluation gaps in a real-time strategy game between human and artificial intelligence players. IEEE Access, 6, 13575–13586.
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems (pp. 1097–1105).
Lee, A. X., Nagabandi, A., Abbeel, P., & Levine, S. (2019). Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. arXiv preprint arXiv:1907.00953.
Li, Y. (2017). Deep reinforcement learning: An overview. arXiv preprint arXiv:1701.07274
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., ... Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971.
Littman, M. L. (2015). Reinforcement learning improves behaviour from evaluative feedback. Nature, 521(7553), 445–451.
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, O. P., & Mordatch, I. (2017). Multi-agent actor-critic for mixed cooperative-competitive environments. In Advances in Neural Information Processing Systems (pp. 6379–6390).
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., ... Kavukcuoglu, K. (2016). Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning (pp. 1928–1937).
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., ... Petersen, S. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529
Nair, A., Srinivasan, P., Blackwell, S., Alcicek, C., Fearon, R., De Maria, A., ... Legg, S. (2015). Massively parallel methods for deep reinforcement learning. arXiv preprint arXiv:1507.04296.
Ontanón, S., Synnaeve, G., Uriarte, A., Richoux, F., Churchill, D., & Preuss, M. (2013). A survey of real-time strategy game AI research and competition in StarCraft. IEEE Transactions on Computational Intelligence and AI in Games, 5(4), 293–311.
Peng, P., Yuan, Q., Wen, Y., Yang, Y., Tang, Z., Long, H., & Wang, J. (2017). Multiagent bidirectionally-coordinated nets for learning to play starcraft combat games. arXiv preprint arXiv:1703.10069, 2.
Riedmiller, M., Gabel, T., Hafner, R., & Lange, S. (2009). Reinforcement learning for robot soccer. Autonomous Robots, 27(1), 55–73.
Russell, S. J., & Norvig, P. (2016). Artificial intelligence: A modern approach. Malaysia: Pearson Education Limited.
Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2015). Prioritized experience replay. arXiv preprint arXiv:1511.05952.
Shantia, A., Begue, E., & Wiering, M. (2011, July). Connectionist reinforcement learning for intelligent unit micro management in starcraft. In The 2011 International Joint Conference on Neural Networks (pp. 1794–1801). IEEE.
Shao, K., Zhu, Y., & Zhao, D. (2017). Cooperative reinforcement learning for multiple units combat in StarCraft. In 2017 IEEE Symposium Series on Computational Intelligence (SSCI) (pp. 1–6). IEEE.
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., ... Dieleman, S. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587), 484
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., ... Lillicrap, T. (2017). Mastering chess and Shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815.
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., & Riedmiller, M. (2014, June). Deterministic policy gradient algorithms.
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., ... Chen, Y. (2017). Mastering the game of go without human knowledge. Nature, 550(7676), 354
Soemers, D. (2014). Tactical planning using MCTS in the game of StarCraft. Doctoral dissertation, Master’s thesis, Department of Knowledge Engineering, Maastricht University) (2014)
Sutton, R. S., Barto, A. G. (2018). Reinforcement learning: An introduction. MIT Press
Synnaeve, G., Nardelli, N., Auvolat, A., Chintala, S., Lacroix, T., Lin, Z., ... Usunier, N. (2016). Torchcraft: A library for machine learning research on real-time strategy games. arXiv preprint arXiv:1611.00625.
Tang, Z., Zhao, D., Zhu, Y., & Guo, P. (2018, June). Reinforcement learning for build-order production in StarCraft II. In 2018 Eighth International Conference on Information Science and Technology (ICIST) (pp. 153–158). IEEE.
Tesauro, G. (1995). Temporal difference learning and TD-Gammon. Communications of the ACM, 38(3), 58–68.
Tesauro, G., Das, R., Chan, H., Kephart, J., Levine, D., Rawson, F., & Lefurgy, C. (2008). Managing power consumption and performance of computing systems using reinforcement learning. In Advances in Neural Information Processing Systems (pp. 1497–1504).
The Brood War API. https://bwapi.github.io/. Last accessed 2020/5/19.
Tian, Y., Gong, Q., Shang, W., Wu, Y., & Zitnick, C. L. (2017). Elf: An extensive, lightweight and flexible research platform for real-time strategy games. In Advances in Neural Information Processing Systems (pp. 2659–2669).
Todorov, E. (2016). MuJoCo: Modeling, simulation and visualization of multi-joint dynamics with contact. Seattle WA: Roboti Publishing.
Uriarte, A., & Ontanón, S. (2015, September). Automatic learning of combat models for RTS games. In Eleventh Artificial Intelligence and Interactive Digital Entertainment Conference (2015, September).
Van Hasselt, H., Guez, A., & Silver, D. (2016, March). Deep reinforcement learning with double q-learning. In Thirtieth AAAI Conference on Artificial Intelligence.
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J. ... Oh, J. (2019). Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 1–5
Vinyals, O., Ewalds, T., Bartunov, S., Georgiev, P., Vezhnevets, A. S., Yeo, M., ... Quan, J. (2017). Starcraft ii: A new challenge for reinforcement learning. arXiv preprint arXiv:1708.04782.
Wang, Z., Bapst, V., Heess, N., Mnih, V., Munos, R., Kavukcuoglu, K., & de Freitas, N. (2016). Sample efficient actor-critic with experience replay. arXiv preprint arXiv:1611.01224.
Wang, Z., Schaul, T., Hessel, M., Van Hasselt, H., Lanctot, M., & De Freitas, N. (2015). Dueling network architectures for deep reinforcement learning. arXiv preprint arXiv:1511.06581.
Wiering, M., & Van Otterlo, M. (2012). Reinforcement learning. Adaptation, Learning, and Optimization, 12, 3.
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., ... Klingner, J. (2016). Google’s neural machine translation system: bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144
Wymann, B., Espié, E., Guionneau, C., Dimitrakakis, C., Coulom, R., & Sumner, A. (2000). Torcs, the open racing car simulator. Software available at https://torcs.sourceforge.net, 4(6).
Xu, S., Kuang, H., Zhi, Z., Hu, R., Liu, Y., & Sun, H. (2019, October). Macro action selection with deep reinforcement learning in Starcraft. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (vol. 15, No. 1, pp. 94–99).
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Ashraf, N.M., Mostafa, R.R., Sakr, R.H., Rashad, M.Z. (2021). A State-of-the-Art Review of Deep Reinforcement Learning Techniques for Real-Time Strategy Games. In: Hamdan, A., Hassanien, A.E., Khamis, R., Alareeni, B., Razzaque, A., Awwad, B. (eds) Applications of Artificial Intelligence in Business, Education and Healthcare . Studies in Computational Intelligence, vol 954. Springer, Cham. https://doi.org/10.1007/978-3-030-72080-3_17
Download citation
DOI: https://doi.org/10.1007/978-3-030-72080-3_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-72079-7
Online ISBN: 978-3-030-72080-3
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)