Abstract
Aiming at the autonomous obstacle avoidance problem of UAV in multi-obstacle map environment, a UAV obstacle avoidance algorithm based on the improved Q learning method is proposed. By analyzing the UAV dynamics principle, the UAV kinematic model is built, and the Markov jump system model is further obtained. Considering the safe distance from the obstacle and the position of the target point, an improved immediate reward function is presented, and a Q learning algorithm of UAV obstacle avoidance is proposed by adopting the \(\varepsilon \)-greedy strategy, which can improve the learning efficiency, realize autonomous obstacle avoidance and optimize the route to the target position. In the simulation experiment, the UAV can track with down different environments and the accumulative rewards are compared and analyzed, which show the effectiveness and advantages of the UAV self-learning algorithm proposed in this paper.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Giordan, D., Adams, M.S., Aicardi, I., et al.: The use of unmanned aerial vehicles (UAVs) for engineering geology applications. Bull. Eng. Geol. Env. 79(7), 3437–3481 (2020)
Guan, H., Sun, X., Su, Y., et al.: UAV-lidar aids automatic intelligent powerline inspection. Int. J. Electr. Power Energy Syst. 130, 106987 (2021)
Sudhakar, S., Vijayakumar, V., Kumar, C.S., et al.: Unmanned Aerial Vehicle (UAV) based Forest Fire Detection and monitoring for reducing false alarms in forest-fires. Comput. Commun. 149, 1–16 (2020)
Dong, J., Ota, K., Dong, M.: UAV-based real-time survivor detection system in post-disaster search and rescue operations. IEEE J. Miniaturization Air Space Syst. 2(4), 209–219 (2021)
Pan, Z., Zhang, C., Xia, Y., et al.: An improved artificial potential field method for path planning and formation control of the multi-UAV systems. IEEE Trans. Circuits Syst. II Express Briefs 69(3), 1129–1133 (2022)
Wang, H., Yin, P., Zheng, W., Zuo, J.: Mobile robot path planning based on improved A* Algorithm and dynamic window method. ROBOT 42(3), 346–353 (2020)
D’Amato, E., Mattei, M., Notaro, I.: Bi-level flight path planning of UAV formations with collision avoidance. J. Intell. Rob. Syst. 93(1–2), 193–211 (2019)
Qi, J., Yang, H., Sun, H.: MOD-RRT*: a sampling-based algorithm for robot path planning in dynamic environment. IEEE Trans. Industr. Electron. 68(8), 7244–7251 (2020)
Ma, Z., Wang, C., Niu, Y., et al.: A saliency-based reinforcement learning approach for a UAV to avoid flying obstacles. Robot. Auton. Syst. 100, 108–118 (2018)
Wang, C., Wang, J., Shen, Y., et al.: Autonomous navigation of UAVs in large-scale complex environments: a deep reinforcement learning approach. IEEE Trans. Veh. Technol. 68(3), 2124–2136 (2019)
Wang, Y., He, H., Sun, C.: Learning to navigate through complex dynamic environment with modular deep reinforcement learning. IEEE Trans. Games 10(4), 400–412 (2018)
Duo, N., Lu, Q., Lin, H., Wei, H.: Step into high-dimensional and continuous action space: a survey on applications of deep reinforcement learning to robotics. ROBOT 41(2), 276–288 (2019)
Huang, Q.: Model-based or model-free, a review of approaches in reinforcement learning. In: Proceedings of the 2020 International Conference on Computing and Data Science (CDS), pp. 219–221. IEEE (2020)
Saxena, V., Jaldén, J., Klessig, H.: Optimal UAV base station trajectories using flow-level models for reinforcement learning. IEEE Trans. Cogn. Commun. Netw. 5(4), 1101–1112 (2019)
Hoel, C.J., Driggs-Campbell, K., Wolff, K., et al.: Combining planning and deep reinforcement learning in tactical decision making for autonomous driving. IEEE Trans. Intell. Veh. 5(2), 294–305 (2019)
Speck, C., Bucci, D.J.: Distributed UAV swarm formation control via object-focused, multi-objective SARSA. In: Proceedings of the 2018 Annual American Control Conference (ACC), pp. 6596–6601. IEEE (2018)
Low, E.S., Ong, P., Cheah, K.C.: Solving the optimal path planning of a mobile robot using improved Q-learning. Robot. Auton. Syst. 115, 143–161 (2019)
Wang, X., Chen, H., Zhao, S.: Formation control of large-scale fixed-wing unmanned aerial vehicle swarms. Control Decis. 36(9), 2063–2073 (2021)
Jang, B., Kim, M., Harerimana, G., et al.: Q-learning algorithms: a comprehensive classification and applications. IEEE Access 7, 133653–133667 (2019)
Blasi, L., D’Amato, E., Mattei, M., et al.: Path planning and real-time collision avoidance based on the essential visibility graph. Appl. Sci. 10(16), 5613 (2020)
Pan, Y., Dong, Y., Wang, D., et al.: Three-dimensional reconstruction of structural surface model of heritage bridges using UAV-based photogrammetric point clouds. Remote Sens. 11(10), 1204 (2019)
Tsardoulias, E.G., Iliakopoulou, A., Kargakos, A., et al.: A review of global path planning methods for occupancy grid maps regardless of obstacle density. J. Intell. Rob. Syst. 84(1), 829–858 (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 Beijing HIWING Sci. and Tech. Info Inst
About this paper
Cite this paper
Gao, H., Li, J. (2023). Multi-obstacle Avoidance of UAV Based on Improved Q Learning Algorithm. In: Fu, W., Gu, M., Niu, Y. (eds) Proceedings of 2022 International Conference on Autonomous Unmanned Systems (ICAUS 2022). ICAUS 2022. Lecture Notes in Electrical Engineering, vol 1010. Springer, Singapore. https://doi.org/10.1007/978-981-99-0479-2_6
Download citation
DOI: https://doi.org/10.1007/978-981-99-0479-2_6
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-0478-5
Online ISBN: 978-981-99-0479-2
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)