面向车联网的多智能体强化学习边云协同卸载

doi:10.19678/j.issn.1000-3428.0058323

计算机工程 ›› 2021, Vol. 47 ›› Issue (4): 13-20. doi: 10.19678/j.issn.1000-3428.0058323

面向车联网的多智能体强化学习边云协同卸载

叶佩文¹, 贾向东², 杨小蓉¹, 牛春雨¹

1. 西北师范大学计算机科学与工程学院, 兰州 730070;
2. 南京邮电大学江苏省无线通信重点实验室, 南京 214215

收稿日期:2020-05-14 修回日期:2020-06-26 发布日期:2020-07-17
作者简介:叶佩文(1993-),男,硕士研究生,主研方向为车联网通信、移动边缘计算;贾向东,教授、博士;杨小蓉、牛春雨,硕士研究生。
基金资助:
国家自然科学基金（61861039，61561043，61261015）；甘肃省科技计划“无人机关键技术研究”（18YF1GA060）。

Collaborative Edge and Cloud Offloading for Internet of Vehicles Using Multi-Agent Reinforcement Learning

YE Peiwen¹, JIA Xiangdong², YANG Xiaorong¹, NIU Chunyu¹

1. College of Computer Science and Engineering, Northwest Normal University, Lanzhou 730070, China;
2. Wireless Communication Key Lab of Jiangsu Province, Nanjing University of Posts and Telecommunications, Nanjing 214215, China

Received:2020-05-14 Revised:2020-06-26 Published:2020-07-17

摘要/Abstract

摘要： 车联网边缘计算是实现车联网系统低时延和高可靠性的关键技术，但现有方法普遍存在场景趋同和系统建模局限的问题，同时包含复杂的训练过程并面临维灾风险。通过结合云计算技术，提出一种基于多智能体强化学习的边云协同卸载方案。依据随机几何理论计算卸载节点覆盖概率，对车辆节点与卸载对象进行预配对。利用线性Q函数分解方法反映每个智能体多效用因子与任务决策间的映射关系，通过云端协同机制将智能体决策记录作为经验上传到云端，并在云端将训练更完备的神经网络反馈到边缘节点。仿真结果表明，该方案在功耗和延时方面性能优于单一固定边缘的计算策略，且算法复杂度较低，能够有效提升边云协同卸载能力，实现低时延、高可靠的任务卸载。

关键词: 车联网, 多智能体强化学习, 随机几何理论, 边云协同计算, 任务卸载策略, 资源分配

Abstract: Edge computing for Internet of Vehicles(IoV) is key to realizing highly reliable and low-latency IoV systems.However,existing methods generally have the problems of scene convergence and system modeling limitations,and are faced with complex training processes and disaster maintenance risks.By combining the cloud computing technology,this paper proposes a collaborative edge and cloud offloading scheme based on multi-agent reinforcement learning.The strategy uses the stochastic geometry theory to calculate the coverage probability of the offloading nodes and pre-match the vehicular nodes to offloading objects.On this basis,the linear Q function decomposition method is used to reflect the mapping relationship between each agent's multi-utility factor and task decision.Then through the collaborative cloud and edge computing mech anism,each agent's decision records are uploaded to the cloud as experience,and the more comprehensively trained neural network is returned to the edge nodes.The results of simulation show that the proposed scheme outperforms the computing strategies using only fixed edge servers in terms of power consumption and latency.The method reduces the algorithm complexity,and can significantly improve the collaborative edge and cloud offloading ability to realize highly reliable and low-latency task offloading.

Key words: Internet of Vehicles(IoV), multi-agent reinforcement learning, stochastic geometry theory, collaborative edge and cloud computing, task offloading strategy, resource allocation

中图分类号:

TN929.5

叶佩文, 贾向东, 杨小蓉, 牛春雨. 面向车联网的多智能体强化学习边云协同卸载[J]. 计算机工程, 2021, 47(4): 13-20.

YE Peiwen, JIA Xiangdong, YANG Xiaorong, NIU Chunyu. Collaborative Edge and Cloud Offloading for Internet of Vehicles Using Multi-Agent Reinforcement Learning[J]. Computer Engineering, 2021, 47(4): 13-20.

https://www.ecice06.com/CN/Y2021/V47/I4/13

图/表 7

20210425164215

20210425164218

20210425164220

20210425164223

20210425164227

20210425164230

20210425164233

参考文献

[1] ZHANG L,LIANG Y C,NIYATO D.6G visions:mobile ultra-broadband,super Internet-of-things,and artificial intelligence[J].China Communications,2019,16(8):1-14.
[2] YANG Xiaodong,PEI Xizhen,AN Faying,et al.Message authentication scheme for vehicular ad hoc network using identity-based aggregate signature[J].Computer Engineering,2020,46(2):170-174,182.(in Chinese)杨小东,裴喜祯,安发英,等.基于身份聚合签名的车载自组网消息认证方案[J].计算机工程,2020,46(2):170-174,182.
[3] CHANG Xiang,ZHANG Rongqing,YANG Liuqing.Wireless toward the era of intelligent vehicles[J].IEEE Internet of Things Journal,2019,6(1):188-202.
[4] ZHOU Huan,WANG Hui,CHEN Xi.Data offloading techniques through vehicular ad hoc networks:a survey[J].IEEE Access,2018,6:65250-65259.
[5] YE Peiwen,JIA Xiangdong,YANG Xiaorong,et al.End-to-end physical layer optimization scheme using auto-encoder based on deep learning[J].Computer Engineering,2019,45(12):86-90,97.(in Chinese)叶佩文,贾向东,杨小蓉,等.基于深度学习的自编码器端到端物理层优化方案[J].计算机工程,2019,45(12):86-90,97.
[6] ZHANG Ke,MAO Yuming,LENG Supeng,et al.Mobile-edge computing for vehicular networks:a promising network paradigm with predictive off-loading[J].IEEE Vehicular Technology Magazine,2017,12(2):36-44.
[7] NING Zhaolong,DONG Peiran,WANG Xiaojie,et al.Deep reinforcement learning for intelligent Internet of vehicles:an energy-efficient computational offloading scheme[J].IEEE Transactions on Cognitive Communica-tions and Networking,2019,5(4):1060-1072.
[8] XIE Shuai.A review of deep learning and intensive learning[J].Information Technology & Informatization,2020(5):225-227.(in Chinese)解帅.深度学习和深度强化学习综述[J].信息技术与信息化,2020(5):225-227.
[9] YANG Chao,LIU Yi,CHEN Xin.Efficient mobility aware task offloading for vehicular edge computing networks[J].IEEE Access,2019,7:26652-26664.
[10] CUI Yaping,LIANG Yingjie,WANG Ruyan.Resource allocation algorithm with multi-platform intelligent offloading in D2D-enabled vehicular networks[J].IEEE Access,2019,7:21246-21253.
[11] LI Baozhu,ZHAO Xuhui.New SDN-based architecture for integrated vehicular cloud computing network[C]//Proceedings of 2018 IEEE International Conference on Selected Topics in Mobile and Wireless Networking.Washington D.C.,USA:IEEE Press,2018:1-4.
[12] KING D,ROTSOS C,AGUADO A,et al.The software defined transport network:fundamentals,findings and futures[C]//Proceedings of International Conference on Transparent Optical Networks.Washington D.C.,USA:IEEE Press,2016:1-4.
[13] DAI Y Y,XU D,MAHARJAN S,et al.Artificial intelligence empowered edge computing and caching for Internet of vehicles[J].IEEE Wireless Communications,2019,26(3):12-18.
[14] REJIBA Z,MASIP-BRUIN X.Computation task assignment in vehicular fog computing:a learning approach via neighbor advice[C]//Proceedings of the 18th International Symposium on Network Computing and Applications.Washington D.C.,USA:IEEE Press,2019:1-5.
[15] HOU Xueshi,LI Yong,CHEN Min,et al.Vehicular fog computing:a viewpoint of vehicles as the infrastructures[J].IEEE Transactions on Vehicular Technology,2016,65(6):3860-3873.
[16] ALTHAMARY I,HUANG C W,LIN P.A survey on multi-agent reinforcement learning methods for vehicular networks[C]//Proceedings of the 15th International Wireless Communications and Mobile Computing Conference.Washington D.C.,USA:IEEE Press,2019:1154-1159.
[17] LIU Yi,YU Huimin,XIE Shengli.Deep reinforcement learning for offloading and resource allocation in vehicle edge computing and networks[J].IEEE Transactions on Vehicular Technology,2019,68(11):11158-11168.
[18] YANG H L,XIE X Z,KADOCH M.Intelligent resource management based on reinforcement learning for ultra-reliable and low-latency IoV communication networks[J].IEEE Transactions on Vehicular Technology,2019,68(5):4157-4169.
[19] HE Yin,ZHAO Nan,YIN Hongxi.Integrated networking,caching,and computing for connected vehicles:a deep reinforcement learning approach[J].IEEE Transactions on Vehicular Technology,2018,67(1):44-55.
[20] TAN L T,HU R Q,HANZO L.Twin-timescale artificial intelligence aided mobility-aware edge caching and computing in vehicular networks[J].IEEE Transactions on Vehicular Technology,2019,68(4):3086-3099.
[21] CHEN X F,WU C,ZHANG H G,et al.Decentralized deep reinforcement learning for delay-power tradeoff in vehicular communications[C]//Pro-ceedings of International Conference on Communications.Washington D.C.,USA:IEEE Press,2019:1-6.
[22] CHOI C S,BACCELLI F.Poisson cox point processes for vehicular networks[J].IEEE Transactions on Vehicular Technology,2018,67(10):10160-10165.
[23] CHOI C,BACCELLI F.An analytical framework for coverage in cellular networks leveraging vehicles[J].IEEE Transactions on Communications,2018,66(10):4950-4964.

选择文件类型/文献管理软件名称

选择包含的内容

面向车联网的多智能体强化学习边云协同卸载

Collaborative Edge and Cloud Offloading for Internet of Vehicles Using Multi-Agent Reinforcement Learning

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 7

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	杨小东, 李沐紫, 马国祖, 李松谕, 王彩芬. 车联网中支持非法签名定位的无证书匿名认证方案[J]. 计算机工程, 2024, 50(6): 157-165.
[2]	曾耀平, 夏玉婷, 江伟伟, 刘月强. 加权能耗最小化的无人机辅助移动边缘计算策略研究[J]. 计算机工程, 2024, 50(2): 288-297.
[3]	彭世明, 林士飏, 贾硕, 杨苗会. 基于负载预测的多目标优化任务卸载策略[J]. 计算机工程, 2024, 50(1): 206-215.
[4]	班玉琦, 段利国, 温昊宇, 李爱萍, 赵菊敏. 面向移动感知的计算卸载及资源分配策略研究[J]. 计算机工程, 2023, 49(8): 163-173.
[5]	梅晶, 戴龙宝, 童钊, 邓昕, 王嘉珂. 资源约束下基于Lyapunov优化的自适应卸载算法[J]. 计算机工程, 2023, 49(7): 34-46.
[6]	李强, 仪晋辉, 杜婷婷, 王胜春. 移动边缘计算中基于A3C的依赖任务卸载与资源分配[J]. 计算机工程, 2023, 49(6): 42-52.
[7]	郑丽萍, 赵玉娟, 费选. 基于改进MOEA/D的车联网通信资源分配算法[J]. 计算机工程, 2023, 49(5): 191-197.
[8]	杨立伟, 贾博宇, 王芳, 彭祥原. 可见光通信与WiFi异构网络资源管理算法[J]. 计算机工程, 2023, 49(3): 203-210,220.
[9]	孙扬威, 戚湧. 基于聚类混合采样与PSO-Stacking的车载CAN入侵检测方法[J]. 计算机工程, 2023, 49(1): 138-145.
[10]	牛淑芬, 闫森, 吕锐曦, 周思玮, 张美玲. V2V车联网中隐私保护性异构聚合签密方案[J]. 计算机工程, 2022, 48(9): 20-27,36.
[11]	陈凯, 徐成, 刘宏哲, 代松银. 基于区块链的危险驾驶地图数据评估模型[J]. 计算机工程, 2022, 48(8): 160-165,172.
[12]	吴茂强, 黄旭民, 康嘉文, 余荣. 面向车路协同推断的差分隐私保护研究[J]. 计算机工程, 2022, 48(7): 29-35.
[13]	郑振康, 周金和. 面向多租户网络资源分配的博弈优化策略[J]. 计算机工程, 2022, 48(5): 170-177.
[14]	宋勇春, 王茜竹, 高正念. 基于HAGA的D2D-NOMA资源分配优化算法[J]. 计算机工程, 2022, 48(2): 275-280,290.
[15]	于晶, 鲁凌云, 李翔. 车联网中基于DDQN的边云协作任务卸载机制[J]. 计算机工程, 2022, 48(12): 156-164.

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

面向车联网的多智能体强化学习边云协同卸载

Collaborative Edge and Cloud Offloading for Internet of Vehicles Using Multi-Agent Reinforcement Learning

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 7

参考文献

相关文章 15

编辑推荐

Metrics

本文评价