Autonomous Driving Ramp Merging Model Based on Reinforcement Learning

doi:10.19678/j.issn.1000-3428.0050990

Computer Engineering ›› 2018, Vol. 44 ›› Issue (7): 20-24,31. doi: 10.19678/j.issn.1000-3428.0050990

Special Issue: ；

Previous Articles Next Articles

Autonomous Driving Ramp Merging Model Based on Reinforcement Learning

QIAO Liang ^a,BAO Hong ^a,XUAN Zuxing^a,LIANG Jun ^a,PAN Feng ^b

Autonomous Driving Ramp Merging Model Based on Reinforcement Learning

Received:2018-01-28 Online:2018-07-15 Published:2018-07-15

基于强化学习的无人驾驶匝道汇入模型

乔良^a,鲍泓^a,玄祖兴^a,梁军^a,潘峰^b

传统的强化学习方法受离散状态空间和离散动作空间的限制,不能很好地应用于匝道汇入场景。为此,构建一种基于强化学习的无人驾驶匝道汇入模型。使用深度Q网络构建强化学习模型,依据该模型将匝道汇入问题纳入强化学习问题的范畴后进行求解。实验结果表明,该模型可以针对不同的环境车辆速度采取不同的策略,从而提高无人驾驶在匝道汇入场景下的智能化决策水平。

作者简介:乔良(1991—),男,硕士研究生,主研方向为无人驾驶决策与控制;鲍泓,教授;玄祖兴,副教授;梁军,教授;潘峰,副教授。
基金资助:
国家自然科学基金“视听觉信息的认知计算”重大研究计划重点支持项目“智能车驾驶脑认知技术、平台与转化研究”(91420202);北京市教委科研计划项目(KM201811417006);英国皇家工程院牛顿基金(UK-CIAPP＼324);北京市属高校高水平教师队伍建设支持计划项目 (IDHT20170511)。

Abstract

Abstract:

The traditional reinforcement learning method is limited by discrete state space and discrete action space,and can not be applied to ramp merging scene.Therefore,a reinforcement learning based autonomous driving ramp merging model is constructed.The reinforcement learning model is built by deep Q network.The ramp merging problem is incorporated into the category of reinforcement learning problem and solved.Experimental results show that the model can adopt different strategies for different environment vehicle speeds,thus improving the intelligent decision-making level of the autonomous driving in ramp merging scene.

Key words: autonomous driving, decision-making, ramp merging, reinforcement learning, deep Q network

摘要：

传统的强化学习方法受离散状态空间和离散动作空间的限制,不能很好地应用于匝道汇入场景。为此,构建一种基于强化学习的无人驾驶匝道汇入模型。使用深度Q网络构建强化学习模型,依据该模型将匝道汇入问题纳入强化学习问题的范畴后进行求解。实验结果表明,该模型可以针对不同的环境车辆速度采取不同的策略,从而提高无人驾驶在匝道汇入场景下的智能化决策水平。

关键词: 无人驾驶, 决策, 匝道汇入, 强化学习, 深度Q网络

CLC Number:

TP393

QIAO Liang,BAO Hong,XUAN Zuxing,LIANG Jun,PAN Feng. Autonomous Driving Ramp Merging Model Based on Reinforcement Learning[J]. Computer Engineering, 2018, 44(7): 20-24,31.

乔良,鲍泓,玄祖兴,梁军,潘峰. 基于强化学习的无人驾驶匝道汇入模型[J]. 计算机工程, 2018, 44(7): 20-24,31.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0050990

http://www.ecice06.com/EN/Y2018/V44/I7/20

References

［1］URMSON C,ANHALT J,BAGNELL D,et al.Autonomous driving in urban environments:boss and the urban challenge［J］.Journal of Field Robotics,2008,25(8):425-466.
［2］DONG C,DOLAN J M,LITKOUHI B.Intention estimation for ramp merging control in autonomous driving［C］//Proceedings of 2017 IEEE Intelligent Vehicles Symposium.Washington D.C.,USA:IEEE Press,2017:1584-1589.
［3］HAFNER M R,CUNNINGHAM D,CAMINITI L,et al.Cooperative collision avoidance at intersections:algorithms and experiments ［J］.IEEE Transactions on Intelligent Transportation Systems,2013,14(3):1162-1175.
［4］ALONSO J,MILAN S V,REZ J,et al.Autonomous vehicle control systems for safe crossroads［J］.Transportation Research,Part C:Emerging Technologies,2011,19(6):1095-1110.
［5］MARINESCU D,URN J,BOUROCHE M,et al.On-ramp traffic merging using cooperative intelligent vehicles:a slot-based approach［C］//Proceedings of the 15th IEEE International Conference on Intelligent Transportation Systems.Washington D.C.,USA:IEEE Press,2012:900-906.
(下转第31页) (上接第24页) ［6］WEI J,DOLAN J M,LITKOUHI B.Autonomous vehicle social behavior for highway entrance ramp management［C］//Proceedings of 2013 IEEE Intelligent Vehicles Symposium.Washington D.C.,USA:IEEE Press,2013:201-207.
［7］罗霞,何彪,刘硕智,等.车联网环境下交叉口车辆路径优化控制研究［J］.计算机仿真,2017,34(4):166-171.
［8］HORST R V D,HOGEMA J.Time-to-collision and collision avoidance systems ［EB/OL］.［2018-01-25］.http://www.ictct.org/migrated_2014/ictct_document_nr_365_Horst.pdf.
［9］COSGUN A,MA L,CHIU J,et al.Towards full automated drive in urban environments:a demonstration in gomentum station,California［C］//Proceedings of 2017 IEEE Intelligent Vehicles Symposium.Washington D.C.,USA:IEEE Press,2017:1120- 1128.
［10］ISELE D,COSGUN A,SUBRAMANIAN K,et al.Navigating intersections with autonomous vehicles using deep reinforcement learning［EB/OL］.［2018-01-25］.http://www.doc88.com/p-3813505208748.html.
［11］LITTMAN M L.Reinforcement learning improves beha-viour from evaluative feedback［J］.Nature,2015,521(7553):445-451.
［12］赵冬斌,邵坤,朱圆恒,等.深度强化学习综述:兼论计算机围棋的发展［J］.控制理论与应用,2016,33(6):701-717.
［13］SILVER D,HUANG A,MADDISON C J,et al.Mastering the game of go with deep neural networks and tree search［J］.Nature,2016,529(7587):484-489.
［14］SILVER D,SCHRITTWIESER J,SIMONYAN K,et al.Mastering the game of go without human know-ledge［J］.Nature,2017,550(7676):354-359.
［15］MNIH V,KAVUKCUOGLU K,SILVER D,et al.Playing atari with deep reinforcement learning ［EB/OL］.［2018-01-25］.http://www0.cs.ucl.ac.uk/staff/D.Silver/web/Publications_files/dqn.pdf.
［16］刘全,翟建伟,章宗长,等.深度强化学习综述［J］.计算机学报,2018,41(1):1-27.

[1]	Guanying ZHANG, Peng YI, Dan LI, Di ZHU, Ming MAO. Service Function Chain Deployment Method for Large-Scale Network [J]. Computer Engineering, 2023, 49(8): 122-129.
[2]	Jing MEI, Longbao DAI, Zhao TONG, Xin DENG, Jiake WANG. Adaptive Offloading Algorithm Based on Lyapunov Optimization Under Resource Constraints [J]. Computer Engineering, 2023, 49(7): 34-46.
[3]	Rong FEI, Mengyang MA, Xiao ZHANG, Xinhong HEI, Qingzheng XU, Yuan QIU. Collision Detection Model for Autonomous Driving Based on Trajectory Prediction and Conflict Detection [J]. Computer Engineering, 2023, 49(7): 10-20.
[4]	Lijiao CAI, Jin QIN, Shuang CHEN. Reinforcement Exploration Method to Keep Away from Old Areas and Avoid Loops [J]. Computer Engineering, 2023, 49(7): 118-124.
[5]	LI Qiang, YI Jinhui, DU Tingting, WANG Shengchun. Dependent Task Offloading and Resource Allocation Based on A3C in Mobile Edge Computing [J]. Computer Engineering, 2023, 49(6): 42-52.
[6]	JIN Zhijun, WANG Hao, FANG Baofu. Multi-Agent Reinforcement Learning Based on Rational Curiosity in Sparse Scenarios [J]. Computer Engineering, 2023, 49(5): 302-309.
[7]	LI Xuesong, ZHANG Qieshi, SONG Chengqun, KANG Yuhang, CHENG Jun. Review of Trajectory Prediction Technology in Autonomous Driving Scenes [J]. Computer Engineering, 2023, 49(5): 1-11.
[8]	RAO Dongning, LUO Nanyue. Stacker Scheduling and Repository Location Recommendation Based on Multi-Task Reinforcement Learning [J]. Computer Engineering, 2023, 49(2): 279-287,295.
[9]	SONG Jian, WANG Zilei. Multi-Goal Multi-Agent Deep Reinforcement Learning Method Based on Value Decomposition [J]. Computer Engineering, 2023, 49(1): 31-40.
[10]	WU Rengyu, ZHOU Qiang, YU Hailong, WANG Yasha. Ambulance Dispatching Algorithm in Shenzhen Based on Deep Reinforcement Learning [J]. Computer Engineering, 2022, 48(9): 298-304.
[11]	ZHAO Yinfu, FENG Zhengyong. Fast Training Method for Manipulator Control Based on Deep Reinforcement Learning [J]. Computer Engineering, 2022, 48(8): 113-120.
[12]	DU Tian, LI Xin, LAI Chengzhe, ZHENG Dong. Secure Trust Management Scheme for Autonomous Driving Map Updating [J]. Computer Engineering, 2022, 48(6): 154-166.
[13]	LI Zifan, WANG Hao, FANG Baofu. A Method for Multi-Agent Cooperation Based on Multi-Step Dueling Network [J]. Computer Engineering, 2022, 48(5): 74-81.
[14]	ZHOU Ruipeng, QIN Jin. Reinforcement Exploration Strategy Based on Best Sub-Strategy Memory [J]. Computer Engineering, 2022, 48(2): 106-112.
[15]	CUI Jingyang, CHEN Zhenguo, TIAN Liqin, ZHANG Guanghua. Overview of User and Entity Behavior Analytics Technology Based on Machine Learning [J]. Computer Engineering, 2022, 48(2): 10-24.

Please choose a citation manager

Content to export

Autonomous Driving Ramp Merging Model Based on Reinforcement Learning

基于强化学习的无人驾驶匝道汇入模型

PDF

Knowledge

Cited

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

Comments

模态框（Modal）标题

Please choose a citation manager

Content to export

Autonomous Driving Ramp Merging Model Based on Reinforcement Learning

基于强化学习的无人驾驶匝道汇入模型

PDF

Knowledge

Cited

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

Comments