Abstract
Aiming at the problem of difficult convergence of three-dimensional space UAV air game, a deep reinforcement learning air game decision-making algorithm based on improved action strategy is proposed. The main contributions of the paper are as follows: First, a heuristic reward function is designed by introducing situational information such as angle and speed, which alleviates the problem of difficult convergence caused by sparse rewards. Second, an action selection strategy based on short-term advantage is designed to avoid a large number of meaningless repetitions data. Simulation experiments show that the decision-making algorithm based on deep reinforcement learning can avoid risks and occupy a favorable position in different initial situations. The proposed improvement mechanism can effectively improve the convergence efficiency of the algorithm and the success rate of the game.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Raivio, T.: Capture set computation of an optimally guided missile. J. Guid. Control. Dyn. 24(6), 1167–1175 (2001)
Virtanen, K., Raivio, T., Hamalainen, R.P.: Modeling pilot’s sequential maneuvering decisions by a multistage influence diagram. J. Guid. Control Dyn. 27(4), 665–677 (2004)
Wang, R., Gao, Z.: Research on decision system in air game simulation using maneuver library. Flight Dyn. 27(6), 72–75 (2009)
Ernest, N., Cohen, K., Kivelevitch, E., et al.: Genetic fuzzy trees and their application towards autonomous training and control of a squadron of unmanned game aerial vehicles. Unmanned Syst. 3(3), 185–204 (2015)
Huang, C.Q., Dong, K.S., Huang, H.Q., et al.: Autonomous air game maneuver decision using Bayesian inference and moving horizon optimization. J. Syst. Eng. Electron. 29(1), 86–97 (2018)
Ruan, C., Kou, Y., et al.: Research on single step manuvering decision air game based on binary FuzzV comparison method. Command Control Simul. 34(5), 10–13 (2012)
Liu, P., Ma, Y.: A deep reinforcement learning based intelligent decision method for UCAV air combat. In: Mohamed Ali, M.S., Wahid, H., Mohd Subha, N.A., Sahlan, S., Md. Yunus, M.A., Wahap, A.R. (eds.) AsiaSim 2017. CCIS, vol. 751, pp. 274–286. Springer, Singapore (2017). https://doi.org/10.1007/978-981-10-6463-0_24
Mnih, V., Kavukcuoglu, K., Silver, D., et al.: Playing Atari with Deep Reinforcement Learning[EB/OL]. https://arxiv.xilesou.top/pdf/1312.5602 (2013)
Kurniawan, B., Vamplew, P., Papasimeon, M., et al.: An empirical study of reward structures for actor-critic reinforcement learning in air game manoeuvring simulation. In: AI 2019: Advances in Artificial Intelligence, 32nd Australasian Joint Conference, Adelaide, SA, Australia (2019)
Zhou, P., Huang, J.T., Zhang, S., et al.: Research on UAV intelligent air game decision and simulation based on deep reinforcement learning. Acta Aeronautica et Astronautica Sinica 43, 126731 (2022). (in Chinese). https://doi.org/10.7527/S1000-6893.2022.26731
Mao, M.Y., Zhang, A., Zhou, D., et al.: Reinforcement learning of UCAV air game based on maneuver prediction. Electron. Opt. Control 26(2), 5–10 22 (2019)
Kong, W., Zhou, D., Zhao, Y., Yang, W.: Maneuvering strategy generation algorithm for multi-UAV in close-range air game based on deep reinforcement learning and self-play. Control Theory Appl. 39(2), 352–362 (2022)
Hambling, D.: AI outguns a human fighter pilot. New Sci. 247(3297), 12 (2020)
Austin, F., Carbone, G., Falco, M., et al.: Automated maneuvering decisions for air-to-air game. In: Guidance, Navigation and Control Conference, p. 2393. AIAA, Monterey (1987)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 Beijing HIWING Sci. and Tech. Info Inst
About this paper
Cite this paper
Xie, R., Huang, C., Wang, Z., Han, J. (2023). A Deep Reinforcement Learning Algorithm Based on Short-Term Advantage for Air Game Decision-Making. In: Fu, W., Gu, M., Niu, Y. (eds) Proceedings of 2022 International Conference on Autonomous Unmanned Systems (ICAUS 2022). ICAUS 2022. Lecture Notes in Electrical Engineering, vol 1010. Springer, Singapore. https://doi.org/10.1007/978-981-99-0479-2_359
Download citation
DOI: https://doi.org/10.1007/978-981-99-0479-2_359
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-0478-5
Online ISBN: 978-981-99-0479-2
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)