基于二阶时序差分误差的双网络DQN算法

doi:10.19678/j.issn.1000-3428.0054557

计算机工程 ›› 2020, Vol. 46 ›› Issue (5): 78-85,93. doi: 10.19678/j.issn.1000-3428.0054557

基于二阶时序差分误差的双网络DQN算法

陈建平^a,b, 周鑫^a,b, 傅启明^a,b, 高振^a, 付保川^a,b, 吴宏杰^a

苏州科技大学 a. 电子与信息工程学院;b. 江苏省建筑智慧节能重点实验室, 江苏苏州 215009

收稿日期:2019-04-10 修回日期:2019-05-14 发布日期:2019-05-20
作者简介:陈建平(1963-),男,教授,主研方向为建筑节能、智能信息处理;周鑫,硕士研究生;傅启明(通信作者),讲师;高振,副教授;付保川,教授;吴宏杰,副教授。
基金资助:
国家自然科学基金（61772357，61672371）；江苏省重点研发计划项目（BE2017663）；江苏省研究生科研与实践创新计划项目（SJCX18-0881）。

Dual Network DQN Algorithm Based on Second-order Temporal Difference Error

CHEN Jianping^a,b, ZHOU Xin^a,b, FU Qiming^a,b, GAO Zhen^a, FU Baochuan^a,b, WU Hongjie^a

a. School of Electronic and Information Engineering;b. Jiangsu Province Key Laboratory of Intelligent Building Energy Efficiency, Suzhou University of Science and Technology, Suzhou, Jiangsu 215009, China

Received:2019-04-10 Revised:2019-05-14 Published:2019-05-20

摘要/Abstract

摘要： 针对深度Q网络（DQN）算法因过估计导致收敛稳定性差的问题，在传统时序差分（TD）的基础上提出N阶TD误差的概念，设计基于二阶TD误差的双网络DQN算法。构造基于二阶TD误差的值函数更新公式，同时结合DQN算法建立双网络模型，得到两个同构的值函数网络分别用于表示先后两轮的值函数，协同更新网络参数，以提高DQN算法中值函数估计的稳定性。基于Open AI Gym平台的实验结果表明，在解决Mountain Car和Cart Pole问题方面，该算法较经典DQN算法具有更好的收敛稳定性。

关键词: 深度强化学习, 马尔科夫决策过程, 深度Q网络, 二阶时序差分误差, 梯度下降

Abstract: Aiming at the problem of poor convergence stability caused by overestimation of Depth Q-Network(DQN) algorithm,on the basis of traditional Temporal Difference(TD),the concept of n-order TD error is proposed and a dual-network DQN algorithm based on second-order TD error is designed.A value function updating formula based on second-order TD error is constructed.Meanwhile,a two-network model is established in combination with DQN algorithm,and two isomorphic value function networks are obtained,which are respectively used to represent the value functions of two successive rounds,and the network parameters are cooperatively updated to improve the stability of value function estimation in DQN algorithm.Experimental results based on the Open AI Gym platform show that,the proposed algorithm has better convergence stability compared with the classical DQN algorithm in solving the Mountain Car and Cart Pole problems.

Key words: Deep Reinforcement Learning(DRL), Markov Decision Process(MDP), Deep Q-Network(DQN), second-order Temporal Difference(TD) error, gradient descent

中图分类号:

TP391

陈建平, 周鑫, 傅启明, 高振, 付保川, 吴宏杰. 基于二阶时序差分误差的双网络DQN算法[J]. 计算机工程, 2020, 46(5): 78-85,93.

CHEN Jianping, ZHOU Xin, FU Qiming, GAO Zhen, FU Baochuan, WU Hongjie. Dual Network DQN Algorithm Based on Second-order Temporal Difference Error[J]. Computer Engineering, 2020, 46(5): 78-85,93.

https://www.ecice06.com/CN/Y2020/V46/I5/78

图/表 9

20200513201116

20200513201122

20200513201128

20200513201133

20200513201137

20200513201141

20200513201147

20200513201152

20200513201157

参考文献

[1] SUTTON R S,BARTO A G.Reinforcement learning:an introduction[M].Cambridge,USA:MIT Press,2018.
[2] MAO Jiangyun,WU Hao,SUN Weiwei.Vehicle trajectory anomaly detection in road network via Markov decision process[J].Chinese Journal of Computers,2018,41(8):1928-1942.(in Chinese)毛江云,吴昊,孙未未.路网空间下基于马尔可夫决策过程的异常车辆轨迹检测算法[J].计算机学报,2018,41(8):1928-1942.
[3] YAN Qicui.Research on the method to solve the dimension disaster problem in reinforcement learning[D].Suzhou:Soochow University,2010.(in Chinese)闫其粹.解决强化学习中维数灾问题的方法研究[D].苏州:苏州大学,2010.
[4] HINTON G E,SALAKHUTDINOV R R.Reducing the dimensionality of data with neural networks[J].Science,2006,313(5786):504-507.
[5] CHENG Jiayuan.Application framework of deep learning in radar communication object recognition[J].Modern Radar,2018,40(8):55-59.(in Chinese)程嘉远.深度学习在雷达通信目标识别中的应用框架[J].现代雷达,2018,40(8):55-59.
[6] DENG Li,YU Dong.Deep learning:methods and applications[M].Hanover,USA:Now Publishers Inc.,2014.
[7] MOUSAVI S S,SCHUKAT M,HOWLEY E.Deep reinforcement learning:an overview[C]//Proceedings of SAI Intelligent Systems Conference.Berlin,Germany:Springer,2017:426-440.
[8] LIU Quan,ZHAI Jianwei,ZHANG Zongchang,et al.A survey on deep reinforcement learning[J].Chinese Journal of Computers,2018,41(1):1-27.(in Chinese)刘全,翟建伟,章宗长,等.深度强化学习综述[J].计算机学报,2018,41(1):1-27.
[9] SILVER D,HUANG A,MADDISON C J,et al.Mastering the game of go with deep neural networks and tree search[J].Nature,2016,529(7587):484-489.
[10] WANG Z,SCHAUL T,HESSEL M,et al.Dueling network architectures for deep reinforcement learning[EB/OL].[2019-04-10].https://arxiv.org/pdf/1511.06581.pdf.
[11] LEVINE S,FINN C,DARRELL T,et al.End-to-end training of deep visuomotor policies[J].Journal of Machine Learning Research,2015,17(1):1-40.
[12] LEVINE S,PASTOR P,KRIZHEVSKY A,et al.Learning hand-eye coordination for robotic grasping with large-scale data collection[C]//Proceedings of International Symposium on Experimental Robotics.Berlin,Germany:Springer,2016:173-184.
[13] SHIBATA K,IIDA M.Acquisition of box pushing by direct-vision-based reinforcement learning[C]//Proceedings of the Society of Instrument and Control Engineers Annual Conference.Washington D.C.,USA:IEEE Press,2003:2322-2327.
[14] LANGE S,RIEDMILLER M.Deep auto-encoder neural networks in reinforcement learning[C]//Proceedings of International Joint Conference on Neural Networks.Washington D.C.,USA:IEEE Press,2010:1-8.
[15] KOUTNIK J,SCHMIDHUBER J,GOMEZ F.Online evolution of deep convolutional network for vision-based reinforcement learning[C]//Proceedings of International Conference on Simulation of Adaptive Behavior.Berlin,Germany:Springer,2014:260-269.
[16] ABTAHI F,ZHU Z,BURRY A M.A deep reinforcement learning approach to character segmentation of license plate images[C]//Proceedings of International Conference on Machine Vision Applications.Washington D.C.,USA:IEEE Press,2015:539-542.
[17] LIAO Xiaomin,YAN Shaohu,SHI Jia,et al.Deep reinforcement learning based resource allocation algorithm in cellular networks[J].Journal on Communications,2019,40(2):11-18.(in Chinese)廖晓闽,严少虎,石嘉,等.基于深度强化学习的蜂窝网资源分配算法[J].通信学报,2019,40(2):11-18.
[18] MNIH V,KAVUKCUOGLU K,SILVER D,et al.Playing Atari with deep reinforcement learning[EB/OL].[2019-04-10].https://arxiv.org/pdf/1312.5602.pdf.
[19] WATKINS C J C H.Learning from delayed rewards[J].Robotics and Autonomous Systems,1989,15(4):233-235.
[20] MNIH V,KAVUKCUOGLU K,SILVER D,et al.Human-level control through deep reinforcement learning[J].Nature,2015,518(7540):529-533.
[21] BAI Chenjia,LIU Peng,ZHAO Wei,et al.Active sampling for deep Q-learning based on TD-error adaptive correction[J].Journal of Computer Research and Development,2019,56(2):262-280.(in Chinese)白辰甲,刘鹏,赵巍,等.基于TD-error自适应校正的深度Q学习主动采样方法[J].计算机研究与发展,2019,56(2):262-280.
[22] LIU Quan,ZHAI Jianwei,ZHONG Shan,et al.A deep recurrent Q-network based on visual attention mechanism[J].Chinese Journal of Computers,2017,40(6):127-140.(in Chinese)刘全,翟建伟,钟珊,等.一种基于视觉注意力机制的深度循环Q网络模型[J].计算机学报,2017,40(6):127-140.

选择文件类型/文献管理软件名称

选择包含的内容

基于二阶时序差分误差的双网络DQN算法

Dual Network DQN Algorithm Based on Second-order Temporal Difference Error

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 9

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	张俊娜, 李天泽, 赵晓焱, 袁培燕. 一种基于DQN的去中心化优先级卸载策略[J]. 计算机工程, 2024, 50(9): 235-245.
[2]	石琼, 段辉, 师智斌. 基于深度强化学习的可信任务卸载方案[J]. 计算机工程, 2024, 50(8): 142-152.
[3]	孙文洁, 李宗民, 孙浩淼. 基于图神经网络的多智能体强化学习值函数分解方法[J]. 计算机工程, 2024, 50(5): 62-70.
[4]	傅明建, 郭福强. 基于深度强化学习的无信号灯路口决策研究[J]. 计算机工程, 2024, 50(5): 91-99.
[5]	杜海军, 余粟. 基于时空图注意力网络的服务机器人动态避障[J]. 计算机工程, 2024, 50(2): 105-112.
[6]	蔡梓越, 谭北海, 余荣, 黄旭民, 王思明. 面向6G物联网设备协同的区块链动态分片[J]. 计算机工程, 2024, 50(1): 50-59.
[7]	沈志东, 岳恒宪. 基于分布式扰动的文本对抗训练方法[J]. 计算机工程, 2023, 49(9): 16-22.
[8]	孔凌辉, 饶哲恒, 徐彦彦, 潘少明. 基于深度强化学习的无线网络智能路由算法[J]. 计算机工程, 2023, 49(9): 199-207, 216.
[9]	胡水. 基于深度强化学习的智能兵棋推演决策方法[J]. 计算机工程, 2023, 49(9): 303-312.
[10]	张冠莹, 伊鹏, 李丹, 朱棣, 毛明. 面向大规模网络的服务功能链部署方法[J]. 计算机工程, 2023, 49(8): 122-129.
[11]	梅晶, 戴龙宝, 童钊, 邓昕, 王嘉珂. 资源约束下基于Lyapunov优化的自适应卸载算法[J]. 计算机工程, 2023, 49(7): 34-46.
[12]	蔡丽娇, 秦进, 陈双. 远离旧区域和避免回路的强化探索方法[J]. 计算机工程, 2023, 49(7): 118-124.
[13]	李强, 仪晋辉, 杜婷婷, 王胜春. 移动边缘计算中基于A3C的依赖任务卸载与资源分配[J]. 计算机工程, 2023, 49(6): 42-52.
[14]	饶东宁, 罗南岳. 基于多任务强化学习的堆垛机调度与库位推荐[J]. 计算机工程, 2023, 49(2): 279-287,295.
[15]	李奇儒, 耿霞. 基于改进DQN算法的机器人路径规划[J]. 计算机工程, 2023, 49(12): 111-120.

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

基于二阶时序差分误差的双网络DQN算法

Dual Network DQN Algorithm Based on Second-order Temporal Difference Error

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 9

参考文献

相关文章 15

编辑推荐

Metrics

本文评价