Trusted Task Offloading Scheme Based on Deep Reinforcement Learning

doi:10.19678/j.issn.1000-3428.0069352

Abstract

Abstract:

To address security concerns related to the trust worthiness of edge servers in Mobile Edge Computing (MEC) as well as the challenges of slow convergence and significant fluctuations in task offloading schemes based on Deep Reinforcement Learning(DRL), this study proposes a task offloading scheme based on trust perception and the DRL algorithm. A multisource feedback trust fusion model is first constructed that utilizes the combined weighting of objective information entropy and historical offload times to assess edge server credibility. The Priority Experience Replay(PER)-SAC algorithm, based on priority experience sampling, is then used to treat the base station as an intelligent agent responsible for offloading decisions in computational tasks. Experimental results show that the proposed scheme has superior performance and convergence as compared with the TASACO, SRTO-DDPG, and I-PPO schemes. The cumulative reward, delay, and energy consumption indicators of the proposed scheme all exhibit optimality with a faster convergence speed and minimized fluctuation range. Compared with the TASACO solution under various test scenarios, the energy consumption performance of the proposed scheme is improved by at least 5.8%, with a maximum improvement of 32.2%, and the latency performance is improved by at least 8.5%, with a maximum improvement of 21.3%.

Key words: Mobile Edge Computing(MEC), task offloading, network security, Deep Reinforcement Learning(DRL), trust mechanism

摘要：

针对移动边缘计算(MEC)中边缘服务器是否可信的安全性问题, 以及基于深度强化学习(DRL)的任务卸载方案存在收敛慢、波动大的难题, 提出一种基于信任感知和DRL算法的任务卸载方案。首先, 构建基于客观信息熵和历史卸载次数组合赋权的多源反馈信任融合模型, 用于聚合信任反馈信息, 评估边缘服务器的可信度; 然后, 利用基于优先级经验采样的优先经验回放(PER)-SAC算法, 将基站作为智能体, 负责计算任务的卸载决策。实验结果表明, 该方案相较于TASACO、SRTO-DDPG和I-PPO方案, 具有更优的性能和更好的收敛性, 其累积奖励、时延和能耗指标均为最优, 且其收敛速度更快、波动幅度更小, 在多个测试场景下, 相较于TASACO方案能耗性能最少提升5.8%, 最大提升32.2%, 时延性能最少提升8.5%, 最大提升21.3%。

关键词: 移动边缘计算, 任务卸载, 网络安全, 深度强化学习, 信任机制

Qiong SHI, Hui DUAN, Zhibin SHI. Trusted Task Offloading Scheme Based on Deep Reinforcement Learning[J]. Computer Engineering, 2024, 50(8): 142-152.

石琼, 段辉, 师智斌. 基于深度强化学习的可信任务卸载方案[J]. 计算机工程, 2024, 50(8): 142-152.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0069352

https://www.ecice06.com/EN/Y2024/V50/I8/142

Figures/Tables 8

Fig.1 A network model of mobile edge computing task offloading

Fig.2 Task offloading process based on deep reinforcement learning and trust perception

Fig.3 Convergence curves of different algorithms

Fig.4 Comparison of global convergence time

Fig.5 Comparison of task failure ratios

Fig.6 Time latency curve

Fig.7 Energy consumption curve

References 25

1	SADATDIYNOV K, CUI L Z, ZHANG L, et al. A review of optimization methods for computation offloading in edge computing networks. Digit Commun Networks, 2022, 9, 450- 461.
2	班玉琦, 段利国, 温昊宇, 等. 面向移动感知的计算卸载及资源分配策略研究. 计算机工程, 2023, 49(8): 163- 173. doi: 10.19678/j.issn.1000-3428.0066522
	BAN Y Q, DUAN L G, WEN H Y, et al. Research on mobility-aware computation offloading and resource allocation strategy. Computer Engineering, 2023, 49(8): 163- 173. doi: 10.19678/j.issn.1000-3428.0066522
3	MANSOURI Y, ALI BABAR M. A review of edge computing: features and resource virtualization. Journal of Parallel and Distributed Computing, 2021, 150, 155- 183. doi: 10.1016/j.jpdc.2020.12.015
4	李小平, 周志星, 陈龙, 等. 异构边缘资源的任务卸载和协同调度. 计算机研究与发展, 2023, 60(6): 1296- 1307. URL
	LI X P, ZHOU Z X, CHEN L, et al. Task offloading and cooperative scheduling for heterogeneous edge resources. Journal of Computer Research and Development, 2023, 60(6): 1296- 1307. URL
5	黄腾飞, 杜永文, 刘帅, 等. 边缘计算中时延敏感的启发式任务卸载方法[J/OL]. 哈尔滨工业大学学报, 1-12[2024-01-02]. https://kns.cnki.net/kcms/detail/23.1235.T.20231214.1310.013.html.
	HUANG T F, DU Y W, LIU S, et al. Latency-sensitive heuristic task offloading method in edge computing[J/OL]. Journal of Harbin Institute of Technology, 1-12[2024-01-02]. https://kns.cnki.net/kcms/detail/23.1235.T.20231214.1310.013.html. (in Chinese)
6	WU H M, CHEN J Q, NGUYEN T N, et al. Lyapunov-guided delay-aware energy efficient offloading in ⅡoT-MEC systems. IEEE Transactions on Industrial Informatics, 2023, 19(2): 2117- 2128. doi: 10.1109/TII.2022.3206787
7	许斌, 赵云凯, 朱剑鸣, 等. 移动边缘计算不确定性任务持续卸载及资源分配方法. 软件学报, 2024, 35(3): 1466- 1484. URL
	XU B, ZHAO Y K, ZHU J M, et al. Continuous offloading and resource allocation method of uncertain tasks in mobile edge computing. Journal of Software, 2024, 35(3): 1466- 1484. URL
8	彭世明, 林士飏, 贾硕, 等. 基于负载预测的多目标优化任务卸载策略. 计算机工程, 2024, 50(1): 206- 215. doi: 10.19678/j.issn.1000-3428.0066766
	PENG S M, LIN S Y, JIA S, et al. Multi-objective optimization task offloading strategy based on load prediction. Computer Engineering, 2024, 50(1): 206- 215. doi: 10.19678/j.issn.1000-3428.0066766
9	ZHANG H X, YANG Y J, HUANG X Z, et al. Ultra-low latency multi-task offloading in mobile edge computing. IEEE Access, 2021, 9, 32569- 32581. doi: 10.1109/ACCESS.2021.3061105
10	李志华, 余自立. 基于深度强化学习的多用户计算卸载优化模型和算法. 电子与信息学报, 2024, 46(4): 1321- 1332. URL
	LI Z H, YU Z L. A multi-user computation offloading optimization model and algorithm based on deep reinforcement learning. Journal of Electronics & Information Technology, 2024, 46(4): 1321- 1332. URL
11	KESHAVARZNEJAD M, REZVANI M H, ADABI S. Delay-aware optimization of energy consumption for task offloading in fog environments using metaheuristic algorithms. Cluster Computing, 2021, 24(3): 1825- 1853. doi: 10.1007/s10586-020-03230-y
12	赵国生, 王甜甜, 王健. 一种边缘设备动态信任度的评估模型. 计算机工程与科学, 2021, 43(9): 1574- 1583. doi: 10.3969/j.issn.1007-130X.2021.09.007
	ZHAO G S, WANG T T, WANG J. A dynamic trust evaluation model for edge devices. Computer Engineering & Science, 2021, 43(9): 1574- 1583. doi: 10.3969/j.issn.1007-130X.2021.09.007
13	张琳, 魏新艳, 刘茜萍, 等. 基于协作信誉和设备反馈的物联网边缘服务器信任评估算法. 通信学报, 2022, 43(2): 118- 130. URL
	ZHANG L, WEI X Y, LIU X P, et al. Trust evaluation algorithm of IoT edge server based on cooperation reputation and device feedback. Journal on Communications, 2022, 43(2): 118- 130. URL
14	孔文萍. 面向边缘计算的可信资源分配与任务卸载的关键技术研究[D]. 北京: 北京邮电大学, 2022.
	KONG W P. Research on key technologies of trusted resource allocation and task offloading for edge computing[D]. Beijing: Beijing University of Posts and Telecommunications, 2022. (in Chinese)
15	KONG W P, LI X Y, HOU L Y, et al. A reliable and efficient task offloading strategy based on multifeedback trust mechanism for IoT edge computing. IEEE Internet of Things Journal, 2022, 9(15): 13927- 13941. doi: 10.1109/JIOT.2022.3143572
16	LIANG J B, ZHANG M, LEUNG V C M. A reliable trust computing mechanism based on multisource feedback and fog computing in social sensor cloud. IEEE Internet of Things Journal, 2020, 7(6): 5481- 5490. doi: 10.1109/JIOT.2020.2981005
17	李芳, 李东坪. 基于熵权法的组合评价模型. 信息技术与信息化, 2021,(9): 148- 150. doi: 10.3969/j.issn.1672-9528.2021.09.044
	LI F, LI D P. Combination evaluation model based on entropy weight method. Information Technology and Informatization, 2021,(9): 148- 150. doi: 10.3969/j.issn.1672-9528.2021.09.044
18	王倩, 张雅文, 陈思光. 面向边缘计算的智能资源分配与计算迁移研究. 中北大学学报(自然科学版), 2023, 44(1): 48-57, 78. doi: 10.3969/j.issn.1673-3193.2023.01.007
	WANG Q, ZHANG Y W, CHEN S G. Research on intelligent resource allocation and computation offloading mechanism for edge computing. Journal of North University of China (Natural Science Edition), 2023, 44(1): 48-57, 78. doi: 10.3969/j.issn.1673-3193.2023.01.007
19	WANG Y T, SHENG M, WANG X J, et al. Mobile-edge computing: partial computation offloading using dynamic voltage scaling. IEEE Transactions on Communications, 2016, 64(10): 4268- 4282.
20	YUAN J, LI X Y. A reliable and lightweight trust computing mechanism for IoT edge devices based on multi-source feedback information fusion. IEEE Access, 1898, 6, 23626- 23638.
21	梅晶, 戴龙宝, 童钊, 等. 资源约束下基于Lyapunov优化的自适应卸载算法. 计算机工程, 2023, 49(7): 34- 46. doi: 10.19678/j.issn.1000-3428.0067064
	MEI J, DAI L B, TONG Z, et al. Adaptive offloading algorithm based on Lyapunov optimization under resource constraints. Computer Engineering, 2023, 49(7): 34- 46. doi: 10.19678/j.issn.1000-3428.0067064
22	FLEISCHER L, GOEMANS M X, MIRROKNI V S, et al. Tight approximation algorithms for maximum general assignment problems[C]//Proceedings of the 17th annual ACM-SIAM symposium on Discrete algorithm - SODA '06. New York, USA: ACM Press, 2006: 611-620.
23	SCHAUL T, QUAN J, ANTONOGLOU I, et al. Prioritized experience replay[EB/OL]. [2024-01-02]. https://arxiv.org/pdf/1511.05952.
24	李梦菲, 毛莺池, 屠子健, 等. 基于深度确定性策略梯度的服务器可靠性任务卸载策略. 计算机科学, 2022, 49(7): 271- 279. URL
	LI M F, MAO Y C, TU Z J, et al. Server-reliability task offloading strategy based on deep deterministic policy gradient. Computer Science, 2022, 49(7): 271- 279. URL
25	HAARNOJA T, ZHOU A, ABBEEL P, et al. Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor[EB/OL]. [2024-01-02]. http://arxiv.org/abs/1801.01290.

[1]	ZHANG Junna, LI Tianze, ZHAO Xiaoyan, YUAN Peiyan. A Decentralized Priority Offloading Strategy Based on DQN [J]. Computer Engineering, 2024, 50(9): 235-245.
[2]	Jiao CHEN, Yan SHEN. Research on Task Offloading of Mobile Edge Computing for Caching Mechanism [J]. Computer Engineering, 2024, 50(7): 194-203.
[3]	FU Mingjian, GUO Fuqiang. Research on Decision-Making at Intersection Without Traffic Lights Based on Deep Reinforcement Learning [J]. Computer Engineering, 2024, 50(5): 91-99.
[4]	Haijun DU, Su YU. Dynamic Obstacle Avoidance for Service Robots Based on Spatio-Temporal Graph Attention Network [J]. Computer Engineering, 2024, 50(2): 105-112.
[5]	Yaoping ZENG, Yuting XIA, Weiwei JIANG, Yueqiang LIU. Research on UAV-Assisted Mobile Edge Computing Strategy with Weighted Energy Minimization [J]. Computer Engineering, 2024, 50(2): 288-297.
[6]	YANG Jianjun, TANG Dongming, LI Juguang, XIAO Yufeng. Task Offloading Strategy of MEC Based on Improved Artificial Hummingbird Algorithm [J]. Computer Engineering, 2024, 50(10): 291-301.
[7]	Ziyue CAI, Beihai TAN, Rong YU, Xumin HUANG, Siming WANG. Dynamic Blockchain Sharding for 6G Internet of Things Devices Collaboration [J]. Computer Engineering, 2024, 50(1): 50-59.
[8]	Sha ZHOU, Guowei SHEN, Chun GUO. Vulnerability Information Completion Based on Security Knowledge Graph and Reverse Features [J]. Computer Engineering, 2024, 50(1): 145-155.
[9]	Shiming PENG, Shiyang LIN, Shuo JIA, Miaohui YANG. Multi-Objective Optimization Task Offloading Strategy Based on Load Prediction [J]. Computer Engineering, 2024, 50(1): 206-215.
[10]	Linghui KONG, Zheheng RAO, Yanyan XU, Shaoming PAN. Intelligent Routing Algorithm for Wireless Networks Based on Deep Reinforcement Learning [J]. Computer Engineering, 2023, 49(9): 199-207, 216.
[11]	Guanying ZHANG, Peng YI, Dan LI, Di ZHU, Ming MAO. Service Function Chain Deployment Method for Large-Scale Network [J]. Computer Engineering, 2023, 49(8): 122-129.
[12]	Yuqi BAN, Liguo DUAN, Haoyu WEN, Aiping LI, Jumin ZHAO. Research on Mobility-Aware Computation Offloading and Resource Allocation Strategy [J]. Computer Engineering, 2023, 49(8): 163-173.
[13]	Changhong YU, Ya LU, Haixin WANG, Ming GAO. Traffic Classification Algorithm for IoT Device Based on Sliding Time Window [J]. Computer Engineering, 2023, 49(7): 259-268.
[14]	Xingjuan CAI, Yanheng GUO, Tianhao ZHAO, Wensheng ZHANG. Edge Computing Service Deployment and Task Offloading Based on Evolutionary Multitasking [J]. Computer Engineering, 2023, 49(7): 1-9.
[15]	Jing MEI, Longbao DAI, Zhao TONG, Xin DENG, Jiake WANG. Adaptive Offloading Algorithm Based on Lyapunov Optimization Under Resource Constraints [J]. Computer Engineering, 2023, 49(7): 34-46.

Please choose a citation manager

Content to export