基于深度强化学习的无信号灯路口决策研究

doi:10.19678/j.issn.1000-3428.0068112

摘要/Abstract

摘要： 无信号灯左转路口是自动驾驶场景中最为危险的场景之一,如何实现高效安全的左转决策是自动驾驶领域的重大难题。深度强化学习(DRL)算法在自动驾驶决策领域具有广阔应用前景。但是,深度强化学习在自动驾驶场景中存在样本效率低、奖励函数设计困难等问题。提出一种基于专家先验的深度强化学习算法(CBAM-BC SAC)来解决上述问题。首先,利用SMARTS仿真平台获得专家先验知识;然后,使用通道-空间注意力机制(CBAM)改进行为克隆(BC)方法,在专家先验知识的基础上预训练模仿专家策略;最后,使用模仿专家策略指导深度强化学习算法的学习过程,并在无信号灯路口左转决策中进行验证。实验结果表明,基于专家先验的DRL算法比传统的DRL算法更具优势,不仅可以免去人为设置奖励函数的工作量,而且可以显著提高样本效率从而获得更优性能。在无信号灯路口左转场景下,CBAM-BC SAC算法与传统DRL算法(SAC)、基于传统行为克隆的DRL算法(BC SAC)相比,平均通行成功率分别提高了14.2和2.2个百分点。

关键词: 深度强化学习, 自动驾驶, 模仿学习, 行为克隆, 驾驶决策

Abstract: Left-turn intersections without signal lights are among the most dangerous scenes in autonomous driving, and achieving efficient and safe left-turn decision-making is highly challenging in autonomous driving. The Deep Reinforcement Learning(DRL) algorithm has broad prospects in autonomous driving decision-making. However, its sample efficiency is low and it cannot be used to easily design reward functions in autonomous driving. Therefore, a DRL algorithm based on expert priors, abbreviated as CBAM-BC SAC, is proposed to solve the aforementioned problems. First, a Scalable Multiagent RL Training School(SMARTS) simulation platform is used to obtain expert prior knowledge. Subsequently, a Convolutional Block Attention Module(CBAM) is used to improve Behavior Cloning(BC), which pretrains and imitates expert strategies based on the prior knowledge of experts. Finally, the learning process of the DRL algorithm is guided by an imitation expert strategy and verified in a left-turn decision-making at intersection without traffic lights. Experimental results indicate that the DRL algorithm based on expert prior is more advantageous than conventional DRL algorithms. It not only eliminates the workload of manually setting reward functions, but also significantly improves sample efficiency and achieves better performance. In left-turn scene at intersection without traffic lights, the CBAM-BC SAC algorithm improves the average traffic success rate by 14.2 and 2.2 percentage points, respectively, compared with the conventional DRL algorithm SAC and the DRL algorithm BC SAC based on classic BC.

Key words: Deep Reinforcement Learning(DRL), autonomous driving, imitation learning, Behavioral Cloning(BC), driving decision-making

中图分类号:

TP391

傅明建, 郭福强. 基于深度强化学习的无信号灯路口决策研究[J]. 计算机工程, 2024, 50(5): 91-99.

FU Mingjian, GUO Fuqiang. Research on Decision-Making at Intersection Without Traffic Lights Based on Deep Reinforcement Learning[J]. Computer Engineering, 2024, 50(5): 91-99.

https://www.ecice06.com/CN/Y2024/V50/I5/91

参考文献

[1] 刘洪启, 沈涛, 吴云, 等. 非信号控制公路平面交叉口渠化设计方法研究[J]. 公路, 2008, 53(11):164-168. LIU H Q, SHEN T, WU Y, et al. Research on channelization design methods of unsignalized highway at-grade intersection[J]. Highway, 2008, 53(11):164-168.(in Chinese)
[2] 段续庭, 周宇康, 田大新, 等. 深度学习在自动驾驶领域应用综述[J]. 无人系统技术, 2021, 4(6):1-27. DUAN X T, ZHOU Y K, TIAN D X, et al. A review of deep learning applications for autonomous driving[J]. Unmanned Systems Technology, 2021, 4(6):1-27.(in Chinese)
[3] 杨思明, 单征, 丁煜, 等. 深度强化学习研究综述[J]. 计算机工程, 2021, 47(12):19-29. YANG S M, SHAN Z, DING Y, et al. Survey of research on deep reinforcement learning[J]. Computer Engineering, 2021, 47(12):19-29.(in Chinese)
[4] 李雪松, 张锲石, 宋呈群, 等. 自动驾驶场景下的轨迹预测技术综述[J]. 计算机工程, 2023, 49(5):1-11. LI X S, ZHANG Q S, SONG C Q, et al. Review of trajectory prediction technology in autonomous driving scenes[J]. Computer Engineering, 2023, 49(5):1-11.(in Chinese)
[5] 欧阳卓, 周思源, 吕勇, 等. 基于深度强化学习的无信号灯交叉路口车辆控制[J]. 计算机科学, 2022, 49(3):46-51. OUYANG Z, ZHOU S Y, LYU Y, et al. DRL-based vehicle control strategy for signal-free intersections[J]. Computer Science, 2022, 49(3):46-51.(in Chinese)
[6] LI G F, LIN S Y, LI S, et al. Learning automated driving in complex intersection scenarios based on camera sensors:a deep reinforcement learning approach[J]. IEEE Sensors Journal, 2022, 22(5):4687-4696.
[7] KARGAR E, KYRKI V. Vision transformer for learning driving policies in complex multi-agent environments[EB/OL].[2023-06-01]. http://arxiv.org/abs/2109.06514.
[8] 王曙燕, 万顷田. 自动驾驶车辆在无信号交叉口右转驾驶决策技术研究[J]. 计算机应用研究, 2023, 40(5):1468-1472. WANG S Y, WAN Q T. Right-turn driving decisions of autonomous vehicles at signal-free intersections[J]. Application Research of Computers, 2023, 40(5):1468-1472.(in Chinese)
[9] LIU J D, BOYLE L N, BANERJEE A G. An inverse reinforcement learning approach for customizing automated lane change systems[J]. IEEE Transactions on Vehicular Technology, 2022, 71(9):9261-9271.
[10] 高振海, 闫相同, 高菲. 基于逆向强化学习的纵向自动驾驶决策方法[J]. 汽车工程, 2022, 44(7):969-975. GAO Z H, YAN X T, GAO F. A decision-making method for longitudinal autonomous driving based on inverse reinforcement learning[J]. Automotive Engineering, 2022, 44(7):969-975.(in Chinese)
[11] HUANG Z Y, WU J D, LV C. Driving behavior modeling using naturalistic human driving data with inverse reinforcement learning[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(8):10239-10251.
[12] CHEN D, JIANG L S, WANG Y, et al. Autonomous driving using safe reinforcement learning by incorporating a regret-based human lane-changing decision model[C]//Proceedings of the 2020 American Control Conference. Washington D. C., USA:IEEE Press, 2020:4355-4361.
[13] KRASOWSKI H, WANG X, ALTHOFF M. Safe reinforcement learning for autonomous lane changing using set-based prediction[C]//Proceedings of the 23rd International Conference on Intelligent Transportation Systems(ITSC). Washington D. C., USA:IEEE Press, 2020:1-7.
[14] DUAN J L, EBEN LI S, GUAN Y, et al. Hierarchical reinforcement learning for self-driving decision-making without reliance on labelled driving data[J]. IET Intelligent Transport Systems, 2020, 14(5):297-305.
[15] WU J, HUANG Z, HUANG C, et al. Human-in-the-loop deep reinforcement learning with application to autonomous driving[EB/OL].[2023-06-01]. https://arxiv.org/abs/2104.07246v1.
[16] LIANG X D, WANG T R, YANG L N, et al. CIRL:controllable imitative reinforcement learning for vision-based self-driving[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany:Springer, 2018:604-620.
[17] PFEIFFER M, SHUKLA S, TURCHETTA M, et al. Reinforced imitation:sample efficient deep reinforcement learning for mapless navigation by leveraging prior demonstrations[J]. IEEE Robotics and Automation Letters, 2018, 3(4):4423-4430.
[18] CODEVILLA F, SANTANA E, LOPEZ A, et al. Exploring the limitations of behavior cloning for autonomous driving[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Washington D. C., USA:IEEE Press, 2019:9328-9337.
[19] LIU H C, HUANG Z Y, WU J D, et al. Improved deep reinforcement learning with expert demonstrations for urban autonomous driving[C]//Proceedings of the 2022 IEEE Intelligent Vehicles Symposium. Washington D. C., USA:IEEE Press, 2022:921-928.
[20] VEČRÍK M, HESTER T, SCHOLZ J, et al. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards[EB/OL].[2023-06-01]. https://arxiv.org/abs/1707.08817v1.
[21] HUANG Z Y, WU J D, LV C. Efficient deep reinforcement learning with imitative expert priors for autonomous driving[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023, 34(10):7391-7403.
[22] WOO S, PARK J, LEE J Y, et al. CBAM:convolutional block attention module[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany:Springer, 2018:3-19.
[23] SCHAUL T, QUAN J, ANTONOGLOU I, et al. Prioritized experience replay[EB/OL].[2023-06-01]. https://arxiv.org/abs/1511.05952v3.
[24] ZHOU M, LUO J, VILLELA J, et al. SMARTS:scalable multi-agent reinforcement learning training school for autonomous driving[EB/OL].[2023-06-01]. https://arxiv.org/abs/2010.09776v2.
[25] HAARNOJA T, ZHOU A, ABBEEL P, et al. Soft actor-critic:off-policy maximum entropy deep reinforcement learning with a stochastic actor[EB/OL].[2023-06-01]. https://arxiv.org/abs/1801.01290v2.
[26] SCHULMAN J, WOLSKI F, DHARIWAL P, et al. Proximal policy optimization algorithms[EB/OL].[2023-06-01]. https://arxiv.org/abs/1707.06347.

选择文件类型/文献管理软件名称

选择包含的内容