Cloud-Edge Collaborative DNN Inference Based on Deep Reinforcement Learning

doi:10.19678/j.issn.1000-3428.0063579

Abstract

Abstract: In the existing Deep Neural Network(DNN) inference based on cloud-edge collaboration, only the static partition strategy is considered in the homogeneous case of edge devices.However, the influence of the network transmission rate, edge device resources, cloud server load on the optimal partition point of DNN inference computation and the optimal offloading strategy of DNN inference task are not considered in heterogeneous edge device clusters.To solve these problems, this study presents an adaptive DNN inference computation partition and task offloading algorithm based on Deep Reinforcement Learning(DRL).The aim is to minimize the DNN inference delay, and a mathematical model of adaptive task offloading and DNN inference computation partition is established.The state, action space, and reward are defined to transform task offloading and DNN inference computation partition combination optimization problems into the optimal policy problem under the Markov decision process.DRL is used to learn from the experience pool about the approximate optimal strategy of DNN inference computation partition between edge devices and cloud servers and task offloading between heterogeneous edge clusters in a dynamic environment.The experimental results show that compared with several classical DNN inference algorithms, the DNN inference delay of the proposed algorithm in a heterogeneous dynamic environment is reduced by approximately 28.83% on average, proving that the low latency requirement of DNN inference is met in a better manner.

Key words: edge computing, Deep Neural Network(DNN), Deep Reinforcement Learning(DRL), inference computation partition, task offloading

摘要： 现有基于云边协同的深度神经网络（DNN）推理仅涉及边缘设备同构情况下的静态划分策略，未考虑网络传输速率、边缘设备资源、云服务器负载等变化对DNN推理计算最佳划分点的影响，以及异构边缘设备集群间DNN推理任务的最佳卸载策略。针对以上问题，提出基于深度强化学习的自适应DNN推理计算划分和任务卸载算法。以最小化DNN推理时延为优化目标，建立自适应DNN推理计算划分和任务卸载的数学模型。通过定义状态、动作空间和奖励，将DNN推理计算划分和任务卸载组合优化问题转换为马尔可夫决策过程下的最优策略问题。利用深度强化学习方法，从经验池中学习动态环境下边缘设备与云服务器间DNN推理计算划分和异构边缘集群间任务卸载的近似最优策略。实验结果表明，与经典DNN推理算法相比，该算法在异构动态环境下的DNN推理时延约平均降低了28.83%，能更好地满足DNN推理的低时延需求。

关键词: 边缘计算, 深度神经网络, 深度强化学习, 推理计算划分, 任务卸载

CLC Number:

TP391

LIU Xianfeng, LIANG Sai, LI Qiang, ZHANG Jin. Cloud-Edge Collaborative DNN Inference Based on Deep Reinforcement Learning[J]. Computer Engineering, 2022, 48(11): 30-38.

刘先锋, 梁赛, 李强, 张锦. 基于深度强化学习的云边协同DNN推理[J]. 计算机工程, 2022, 48(11): 30-38.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0063579

http://www.ecice06.com/EN/Y2022/V48/I11/30

Figures/Tables 11

References

[1] LIU W B, WANG Z D, LIU X H, et al.A survey of deep neural network architectures and their applications[J].Neurocomputing, 2017, 234:11-26.
[2] 姚相坤, 万里红, 霍宏, 等.基于多结构卷积神经网络的高分遥感影像飞机目标检测[J].计算机工程, 2017, 43(1):259-267. YAO X K, WAN L H, HUO H, et al.Airplane object detection in high resolution remote sensing imagery based on multi-structure convolutional neural network[J].Computer Engineering, 2017, 43(1):259-267.(in Chinese)
[3] YOUNG T, HAZARIKA D, PORIA S, et al.Recent trends in deep learning based natural language processing[J].IEEE Computational Intelligence Magazine, 2018, 13(3):55-75.
[4] POPEL M, TOMKOVA M, TOMEK J, et al.Transforming machine translation:a deep learning system reaches news translation quality comparable to human professionals[J].Nature Communications, 2020, 11(1):1-15.
[5] KANG Y, HAUSWALD J, GAO C, et al.Neurosurgeon:collaborative intelligence between the cloud and mobile edge[J].ACM SIGARCH Computer Architecture News, 2017, 45(1):615-629.
[6] 赵鹏程, 高尚, 于洪梅.基于多智能体深度强化学习的空间众包任务分配[J].吉林大学学报(理学版), 2022, 60(2):321-331. ZHAO P C, GAO J, YU H M.Spatial crowdsourcing task allocation based on multi-agent deep reinforcement learning[J].Journal of Jilin University(Science Edition), 2022, 60(2):321-331.(in Chinese)
[7] 施巍松, 张星洲, 王一帆, 等.边缘计算:现状与展望[J].计算机研究与发展, 2019, 56(1):69-89. SHI W S, ZHANG X Z, WANG Y F, et al.Edge computing:state-of-the-art and future directions[J].Journal of Computer Research and Development, 2019, 56(1):69-89.(in Chinese)
[8] IANDOLA F N, HAN S, MOSKEWICZ M W, et al.SqueezeNet:AlexNet-level accuracy with 50x fewer parameters and <0.5 MB model size[EB/OL].[2022-02-05].https://arxiv.org/abs/1602.07360.
[9] LIU S, LIN Y, ZHOU Z, et al.On-demand deep model compression for mobile devices:a usage-driven model selection framework[C]//Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services.New York, USA:ACM Press, 2018:389-400.
[10] HAN S, MAO H Z, DALLY W J.Deep compression:compressing deep neural networks with pruning, trained quantization and Huffman coding[EB/OL].[2022-02-05].https://arxiv.org/abs/1510.00149.
[11] LI E, ZENG L K, ZHOU Z, et al.Edge AI:on-demand accelerating deep neural network inference via edge computing[J].IEEE Transactions on Wireless Communications, 2020, 19(1):447-457.
[12] TEERAPITTAYANON S, MCDANEL B, KUNG H T.BranchyNet:fast inference via early exiting from deep neural networks[C]//Proceedings of the 23rd International Conference on Pattern Recognition.Washington D.C., USA:IEEE Press, 2016:2464-2469.
[13] ZHAO Z, BARIJOUGH K M, GERSTLAUER A.DeepThings:distributed adaptive deep learning inference on resource-constrained IoT edge clusters[J].IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2018, 37(11):2348-2359.
[14] ZHOU L, SAMAVATIAN M H, BACHA A, et al.Adaptive parallel execution of deep neural networks on heterogeneous edge devices[C]//Proceedings of the 4th ACM/IEEE Symposium on Edge Computing.New York, USA:ACM Press, 2019:195-208.
[15] SIMONYAN K, ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[EB/OL].[2022-02-05].https://arxiv.org/abs/1409.1556.
[16] HUANG L, BI S, ZHANG Y J A.Deep reinforcement learning for online computation offloading in wireless powered mobile-edge computing networks[J].IEEE Transactions on Mobile Computing, 2019, 19(11):2581-2593.
[17] 杨思明, 单征, 丁煜, 等.深度强化学习研究综述[J].计算机工程, 2021, 47(12):19-29. YANG S M, SHAN Z, DING Y, et al.Survey of research on deep reinforcement learning[J].Computer Engineering, 2021, 47(12):19-29.(in Chinese)
[18] 杨锦翔, 熊焰, 黄文超.基于强化学习的安全协议形式化验证优化研究[J].计算机工程, 2021, 47(12):141-146. YANG J X, XIONG Y, HUANG W C.Research on optimization of security protocol formal verification based on reinforcement learning[J].Computer Engineering, 2021, 47(12):141-146.(in Chinese)
[19] ZOU J, HAO T, YU C, et al.A3C-DO:a regional resource scheduling framework based on deep reinforcement learning in edge scenario[J].IEEE Transactions on Computers, 2021, 70(2):228-239.
[20] YAN J, BI S Z, ZHANG Y J A.Offloading and resource allocation with general task graph in mobile edge computing:a deep reinforcement learning approach[J].IEEE Transactions on Wireless Communications, 2020, 19(8):5404-5419.
[21] MOLCHANOV P, TYREE S, KARRAS T, et al.Pruning convolutional neural networks for resource efficient transfer learning[EB/OL].[2022-02-05].https://arxiv.org/abs/1611.06440.
[22] KAN N, ZOU J, TANG K, et al.Deep reinforcement learning-based rate adaptation for adaptive 360-degree video streaming[C]//Proceedings of 2019 IEEE International Conference on Acoustics, Speech and Signal Processing.Washington D.C., USA:IEEE Press, 2019:4030-4034.
[23] MNIH V, KAVUKCUOGLU K, SILVER D, et al.Human-level control through deep reinforcement learning[J].Nature, 2015, 518(7540):529-533.
[24] MIGNON A, ROCHA R L.An adaptive implementation of ε-greedy in reinforcement learning[J].Procedia Computer Science, 2017, 109:1146-1151.
[25] KRIZHEVSKY A, SUTSKEVER I, HINTON G E.ImagNet classification with deep convolutional neural networks[J].Advances in Neural Information Processing Systems, 2012, 25(2):1097-1105.
[26] REDMON J, DIVVALA S, GIRSHICK R, et al.You only look once:unified, real-time object detection[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2016:779-788.

Please choose a citation manager

Content to export