基于QMix的车辆云计算资源动态分配方法

doi:10.19678/j.issn.1000-3428.0063375

摘要/Abstract

摘要： 城市交通智能化和通信技术的进步会产生大量基于车辆的应用，但目前车辆有限的计算资源无法满足车辆应用的计算需求与延迟性约束。车辆云（VC）可以高效地调度资源，从而显著降低任务请求的延迟与传输成本。针对VC环境下任务卸载与计算资源分配问题，提出一个考虑异质车辆和异质任务的计计资源分配算法。对到达的任务构建M/M/1队列模型与计算模型，并定义一个效用函数以最大化系统整体效用。针对环境中车辆地理分布的高度动态系统变化，提出基于双时间尺度的二次资源分配机制（SRA），使用两个不同时间尺度的资源分配决策动作，对其分别构建部分可观测马尔可夫决策过程。两个决策动作通过执行各自的策略获得的奖励进行连接，将问题建模为两层计算资源分配问题。在此基础上提出基于二次资源分配机制的多智能体算法SRA-QMix求解最优策略。仿真结果表明，与深度确定性策略梯度算法对比，该算法的整体效用值和任务完成率分别提高了70%、6%，对于QMix和MADDPG算法分别应用SRA后的任务完成率分别提高了13%与15%，可适用于动态的计算资源分配环境。

关键词: 车辆云, 多智能体强化学习, QMix算法, 任务卸载, 排队理论

Abstract: With the development of urban traffic intelligence and communication technology, several vehicle-based applications can exist.However, the limited computing resources of today's vehicles cannot meet the vehicular applications' computing requirements and latency constraints.The Vehicular Cloudlet (VC) can efficiently dispatch resources to significantly reduce the time delay and transmission cost of the task request.For the task offloading and resource allocation problem in a VC environment, a computing resource allocation algorithm is proposed considering heterogeneous vehicles and tasks.First, M/M/1 queue and computing models are formulated for arriving tasks, and then, a utility function is defined to maximize the overall utility of the system.To deal with a highly dynamic system by vehicle geographical distribution in the environment, a Secondary Resource Allocation (SRA) mechanism based on dual-time-scales is proposed.In this mechanism, two difference time-scales are used for resource allocation decision-making actions and constructing partially observable Markov decision processes.The two decision-making actions are connected through the reward feedbacks obtained by their respective execution strategies, and the problem is modeled as a two-layered computing resource allocation problem.Subsequently, a multi-agent algorithm based on the SRA mechanism, SRA-QMix, is proposed to obtain an optimal strategy.Compared with the deep deterministic policy gradient algorithm, the simulation results show that the proposed algorithm can improve the utility value by 70% and the task finish rate by 6%.In addition, the task finish rate can be improved by 13% and 15% after applying the SRA mechanism for the QMix and MADDPG algorithms.This shows that the scheme based on SRA mechanism can adapt to the dynamic resource allocation environment.

Key words: Vehicular Cloudlet (VC), Multi-Agent Reinforcement Learning(MARL), QMix algorithm, task offloading, queuing theory

中图分类号:

TP391

刘金石, Manzoor Ahmed, 林青. 基于QMix的车辆云计算资源动态分配方法[J]. 计算机工程, 2022, 48(11): 284-290,298.

LIU Jinshi, Manzoor Ahmed, LIN Qing. QMix-Based Method for Dynamic Resource Allocation Leveraging Vehicular Cloudlet Computing[J]. Computer Engineering, 2022, 48(11): 284-290,298.

https://www.ecice06.com/CN/Y2022/V48/I11/284

图/表 7

20230221181836

20230221181840

20230221181844

20230221181847

20230221181850

20230221181854

20230221181857

参考文献

[1] ZHANG K, LENG S P, PENG X, et al.Artificial intelligence inspired transmission scheduling in cognitive vehicular communications and networks[J].IEEE Internet of Things Journal, 2019, 6(2):1987-1997.
[2] LV Z Q, LI J B, DONG C H, et al.Deep learning in the COVID-19 epidemic:a deep model for urban traffic revitalization index[J].Data & Knowledge Engineering, 2021, 135:101912.
[3] 牛瑞彪, 唐伦, 陈婉.小蜂窝云中功率与负载的联合优化分配算法[J].计算机工程, 2017, 43(8):49-55. NIU R B, TANG L, CHEN W.Allocation algorithm of power and load jointing optimization for small cell cloud[J].Computer Engineering, 2017, 43(8):49-55.(in Chinese)
[4] LI B, FEI Z S, CHU Z, et al.Secure transmission for heterogeneous cellular networks with wireless information and power transfer[J].IEEE Systems Journal, 2018, 12(4):3755-3766.
[5] PENG H X, LE L, SHEN X M, et al.Vehicular communications:a network layer perspective[J].IEEE Transactions on Vehicular Technology, 2019, 68(2):1064-1078.
[6] ULLAH S, ABBAS G, ABBAS Z H, et al.RBO-EM:reduced broadcast overhead scheme for emergency message dissemination in VANETs[J].IEEE Access, 2020, 8:175205-175219.
[7] LIN C C, DENG D J.Optimal two-lane placement for hybrid VANET-sensor networks[J].IEEE Transactions on Industrial Electronics, 2015, 62(12):7883-7891.
[8] ABUELELA M, OLARIU S.Taking VANET to the clouds[C]//Proceedings of IEEE MoMMʼ10.Washington D.C., USA:IEEE Press, 2010:2356-2464.
[9] SKONDRAS E, MICHALAS A, VERGADOS D D.Mobility management on 5G vehicular cloud computing systems[J].Vehicular Communications, 2019, 16:15-44.
[10] 董思岐, 吴嘉慧, 李海龙, 等.面向优先级任务的移动边缘计算资源分配方法[J].计算机工程, 2020, 46(3):18-23. DONG S Q, WU J H, LI H L, et al.Resource allocation method for priority task in mobile edge computing[J].Computer Engineering, 2020, 46(3):18-23.(in Chinese)
[11] 唐伦, 胡彦娟, 刘通, 等.移动边缘计算中基于Lyapunov的任务卸载与资源分配算法[J].计算机工程, 2021, 47(3):29-36. TANG L, HU Y J, LIU T, et al.Task offloading and resource allocation algorithm based on Lyapunov in mobile edge computing[J].Computer Engineering, 2021, 47(3):29-36.(in Chinese)
[12] RAZA S, LIU W, AHMED M, et al.An efficient task offloading scheme in vehicular edge computing[J].Journal of Cloud Computing, 2020, 9:28.
[13] JIANG Z Y, ZHOU S, GUO X Y, et al.Task replication for deadline-constrained vehicular cloud computing:optimal policy, performance analysis, and implications on road traffic[J].IEEE Internet of Things Journal, 2018, 5(1):93-107.
[14] SUN F, CHENG N, ZHANG S, et al.Reinforcement learning based computation migration for vehicular cloud computing[C]//Proceedings of 2018 IEEE Global Communications Conference.Washington D.C., USA:IEEE Press, 2018:1-6.
[15] WANG Z, ZHONG Z D, NI M M.Application-aware offloading policy using SMDP in vehicular fog computing systems[C]//Proceedings of 2018 IEEE International Conference on Communications Workshops.Washington D.C., USA:IEEE Press, 2018:511-526.
[16] LIN C C, DENG D J, YAO C C.Resource allocation in vehicular cloud computing systems with heterogeneous vehicles and roadside units[J].IEEE Internet of Things Journal, 2018, 5(5):3692-3700.
[17] NING Z L, DONG P R, WANG X J, et al.Deep reinforcement learning for vehicular edge computing[J].ACM Transactions on Intelligent Systems and Technology, 2019, 10(6):1-24.
[18] VAN HASSELT H, GUEZ A, SILVER D.Deep reinforcement learning with double Q-learning[J].Artificial Intelligence, 2016, 30(1):578-586.
[19] QI Q, WANG J Y, MA Z Y, et al.Knowledge-driven service offloading decision for vehicular edge computing:a deep reinforcement learning approach[J].IEEE Transactions on Vehicular Technology, 2019, 68(5):4192-4203.
[20] LEE S S, LEE S.Resource allocation for vehicular fog computing using reinforcement learning combined with heuristic information[J].IEEE Internet of Things Journal, 2020, 7(10):10450-10464.
[21] LIANG H B, ZHANG X H, HONG X T, et al.Reinforcement learning enabled dynamic resource allocation in the Internet of vehicles[J].IEEE Transactions on Industrial Informatics, 2021, 17(7):4957-4967.
[22] RASHID T, SAMVELYAN M, DE WITT C S, et al.QMIX:monotonic value function factorisation for deep multi-agent reinforcement learning[EB/OL].[2021-10-20].https://arxiv.org/abs/1803.11485.
[23] SHERAZ M, AHMED M, HOU X S, et al.Artificial intelligence for wireless caching:schemes, performance, and challenges[J].IEEE Communications Surveys & Tutorials, 2021, 23(1):631-661.
[24] RAZA S, WANG S G, AHMED M, et al.A survey on vehicular edge computing:architecture, applications, technical issues, and future directions[J].Wireless Communications and Mobile Computing, 2019, 2019:1-19.
[25] SOLAN E, VIEILLE N.Stochastic games[J].Proceedings of the National Academy of Sciences of the United States of America, 2015, 112(45):13743-13746.

选择文件类型/文献管理软件名称

选择包含的内容