Abstract
As autonomous driving moves closer to a real-world application, more and more attention is being paid to the motion planning part of the system. To handle vastness of possible road scenarios, negotiate with other road users and generate an intelligent control strategy in a constantly changing environment, data-driven techniques and artificial intelligence methods seem to be the approach of choice. In this paper, we present reinforcement learning (RL) agent which is embedded in a deterministic, safety envelope. The agent is responsible for generating high-level maneuvers, such as a lane following or a lane change. The primary goal of the agent is to reach a given lane in a given distance, while traveling on a highway. The selected maneuver is then executed with use of deterministic methods utilizing concept of Responsible-Sensitive Safety (RSS) framework, which formalizes safety constrains in a form of mathematical model. The proposed solution has been evaluated in two environments: one in which the agent receives a predefined reward for getting to a correct lane and second, in which it is rewarded for doing this in a time-optimal manner. We have evaluated the proposed solution against an another RL-based agent, which is steering vehicle by low-level control signals, such as acceleration and steering angle.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Althoff, M., Stursberg, O., Buss, M.: Model-based probabilistic collision detection in autonomous driving. IEEE Trans. Intell. Transp. Syst. 10(2), 299–310 (2009). https://doi.org/10.1109/TITS.2009.2018966. http://ieeexplore.ieee.org/document/4895669/
Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L.D., Monfort, M., Muller, U., Zhang, J., Zhang, X., Zhao, J., Zieba, K., et al.: End to End Learning for Self-Driving Cars (2016). http://arxiv.org/abs/1604.07316
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W.: OpenAI Gym. CoRR abs/1606.01540 (2016). http://arxiv.org/abs/1606.01540
Censi, A., Slutsky, K., Wongpiromsarn, T., Yershov, D., Pendleton, S., Fu, J., Frazzoli, E.: Liability, Ethics, and Culture-Aware Behavior Specification using Rulebooks, pp. 8536–8542. Institute of Electrical and Electronics Engineers (IEEE) (2019). https://doi.org/10.1109/icra.2019.8794364
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: An Open Urban Driving Simulator (2017). URL http://arxiv.org/abs/1711.03938
Haarnoja, T., Zhou, A., Abbeel, P., Levine, S.: Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018). http://arxiv.org/abs/1801.01290
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., Levine, S.: Soft Actor-Critic Algorithms and Applications (2018). http://arxiv.org/abs/1812.05905
Hausknecht, M., Stone, P.: Deep recurrent q-learning for partially observable MDPs. In: AAAI Fall Symposium - Technical Report, vol. FS-15-06, pp. 29–37. AI Access Foundation (2015)
Horgan, D., Quan, J., Budden, D., Barth-Maron, G., Hessel, M., van Hasselt, H., Silver, D.: Distributed prioritized experience replay. In: International Conference on Learning Representations (2018). https://openreview.net/forum?id=H1Dy--0Z
Kapturowski, S., Ostrovski, G., Dabney, W., Quan, J., Munos, R.: Recurrent experience replay in distributed reinforcement learning. In: International Conference on Learning Representations (2019). https://openreview.net/forum?id=r1lyTjAqYX
Lopez, P.A., Behrisch, M., Bieker-Walz, L., Erdmann, J., Flotterod, Y.P., Hilbrich, R., Lucken, L., Rummel, J., Wagner, P., Wiebner, E.: Microscopic Traffic Simulation using SUMO. In: IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC, vol. 2018-November, pp. 2575–2582. Institute of Electrical and Electronics Engineers Inc. (2018). https://doi.org/10.1109/ITSC.2018.8569938
Magdici, S., Althoff, M.: Fail-safe motion planning of autonomous vehicles. In: 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), pp. 452–458. IEEE, Rio de Janeiro, Brazil (2016). https://doi.org/10.1109/ITSC.2016.7795594. http://ieeexplore.ieee.org/document/7795594/
Makantasis, K., Kontorinaki, M., Nikolos, I.: A Deep Reinforcement-Learning-based Driving Policy for Autonomous Road Vehicles (2019). http://arxiv.org/abs/1907.05246
Mirchevska, B., Pek, C., Werling, M., Althoff, M., Boedecker, J.: High-level decision making for safe and reasonable autonomous lane changing using reinforcement learning. In: IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC, vol. 2018-November, pp. 2156–2162. Institute of Electrical and Electronics Engineers Inc. (2018). https://doi.org/10.1109/ITSC.2018.8569448
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing Atari with Deep Reinforcement Learning (2013). http://arxiv.org/abs/1312.5602
Nageshrao, S., Tseng, E., Filev, D.: Autonomous Highway Driving using Deep Reinforcement Learning (2019). http://arxiv.org/abs/1904.00035
Ngai, D.C.K., Yung, N.H.C.: A multiple-goal reinforcement learning method for complex vehicle overtaking maneuvers. IEEE Trans. Intell. Transp. Syst. 12, 509–522 (2011). https://doi.org/10.1109/TITS.2011.2106158
Nistér, D., Lee, H.L., Ng, J., Wang, Y.: The Safety Force Field. Technical report
Paden, B., Cap, M., Yong, S.Z., Yershov, D., Frazzoli, E.: A survey of motion planning and control techniques for self-driving urban vehicles. IEEE Trans. Intell. Veh. 1(1), 33–55 (2016). https://doi.org/10.1109/tiv.2016.2578706
Qian, X., de La Fortelle, A., Moutarde, F.: A hierarchical Model Predictive Control framework for on-road formation control of autonomous vehicles. In: 2016 IEEE Intelligent Vehicles Symposium (IV), pp. 376–381. IEEE, Gotenburg, Sweden (2016). https://doi.org/10.1109/IVS.2016.7535413. http://ieeexplore.ieee.org/document/7535413/
Rasekhipour, Y., Khajepour, A., Chen, S.K., Litkouhi, B.: A potential field-based model predictive path-planning controller for autonomous road vehicles. IEEE Trans. Intell. Transp. Syst. 18(5), 1255–1267 (2017). https://doi.org/10.1109/TITS.2016.2604240. http://ieeexplore.ieee.org/document/7576661/
Schaul, T., Quan, J., Antonoglou, I., Silver, D.: Prioritized experience replay. arXiv preprint arXiv:1511.05952 (2015)
Shah, S., Dey, D., Lovett, C., Kapoor, A.: AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles (2017). http://arxiv.org/abs/1705.05065
Shalev-Shwartz, S., Shammah, S., Shashua, A.: On a Formal Model of Safe and Scalable Self-driving Cars (2017). http://arxiv.org/abs/1708.06374
Sutton, R.S., Barto, A.G.: Introduction to Reinforcement Learning, 1st edn. MIT Press, Cambridge (1998)
Talpaert, V., Sobh, I., Ravi Kiran, B., Mannion, P., Yogamani, S., El-Sallab, A., Perez, P.: Exploring applications of deep reinforcement learning for real-world autonomous driving systems. In: VISIGRAPP 2019 - Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, vol. 5, pp. 564–572. SciTePress (2019). https://doi.org/10.5220/0007520305640572
Wang, P., Chan, C.Y.: Formulation of deep reinforcement learning architecture toward autonomous driving for on-ramp merge. In: IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC, vol. 2018-March, pp. 1–6. Institute of Electrical and Electronics Engineers Inc. (2018). https://doi.org/10.1109/ITSC.2017.8317735
Wei, J., Snider, J.M., Gu, T., Dolan, J.M., Litkouhi, B.: A behavioral planning framework for autonomous driving. In: IEEE Intelligent Vehicles Symposium, Proceedings, pp. 458–464. Institute of Electrical and Electronics Engineers Inc. (2014). https://doi.org/10.1109/IVS.2014.6856582
Xu, W., Wei, J., Dolan, J.M., Zhao, H., Zha, H.: A real-time motion planner with trajectory optimization for autonomous vehicles. In: 2012 IEEE International Conference on Robotics and Automation, pp. 2061–2067. IEEE (2012)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Orłowski, M., Wrona, T., Pankiewicz, N., Turlej, W. (2020). Safe and Goal-Based Highway Maneuver Planning with Reinforcement Learning. In: Bartoszewicz, A., Kabziński, J., Kacprzyk, J. (eds) Advanced, Contemporary Control. Advances in Intelligent Systems and Computing, vol 1196. Springer, Cham. https://doi.org/10.1007/978-3-030-50936-1_105
Download citation
DOI: https://doi.org/10.1007/978-3-030-50936-1_105
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-50935-4
Online ISBN: 978-3-030-50936-1
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)