Research on Neural Network Model Generation and Deployment for Edge Intelligence

doi:10.19678/j.issn.1000-3428.0068554

Abstract

Abstract:

The exponential growth of diverse terminal devices, driven by advancements in mobile computing, 5th Generation mobile communication (5G), and Internet of Things (IoT) technologies, has generated vast volumes of semi-structured and unstructured data, as these devices connect to networks. Neural networks are widely applicable because of their unique advantages in mining data. To enhance data processing capabilities and inference accuracy, neural network models are often designed to be complex and thus consume substantial computational resources in both storage and execution. However, edge devices typically have limited computational resources and cannot satisfy the storage and inference requirements of complex neural network models. Therefore, they often rely on cloud computing centers to perform these tasks. This cloud-based collaboration can lead to increased response latency and network bandwidth consumption, incurring potential risks such as the infringement of user privacy through data leakage. To address these issues, this paper introduces Neural network Generation and Deployment (NGD), a method for rapidly generating and deploying tailored neural network models on edge devices, whereby neural network models that match the hardware configuration of edge devices and specific computational task requirements are generated and rapidly deployed onto target devices for local inference. Experiments are conducted on three typical edge devices, and the results demonstrate the effectiveness of NGD in generating and deploying models, affirming its practicality and effectiveness.

Key words: edge intelligence, on-device inference, model generation, automatic deployment, edge device

摘要：

随着移动计算、第五代移动通信技术(5G)以及物联网(IoT)技术的不断演进, 各类终端设备的数量呈现指数级增长。这种激增的终端设备连接到网络产生了巨大的数据流, 对于需要实时处理和快速响应用户任务的需求提出了新的挑战。尤其是在这些海量数据中, 半结构化和非结构化数据所占比例较大, 这使得神经网络因其独特的优势而得到了广泛应用。为了提高数据处理能力和推理精度, 神经网络模型会被设计得非常复杂, 其存储和运行均需要消耗大量的计算资源。然而, 边缘设备通常只配置有限的计算资源, 无法满足存储和运行复杂神经网络模型的需求, 需要借助云计算中心来完成这些任务。这种云协同会引发响应延时和增加网络带宽消耗, 并带来用户隐私数据泄露等潜在风险。为了解决这些问题, 提出一种面向边缘智能的神经网络模型快速生成与自动化部署(NGD)方法, 根据边缘设备的硬件配置和承载的具体计算任务需求, 生成与其匹配的神经网络模型, 并将其快速部署在目标设备上, 实现设备本地推理。在3种典型的硬件平台上的神经网络模型生成与部署实验结果表明, NGD方法能够高效地为资源受限的边缘设备生成匹配的神经网络模型, 并快速地将其部署在设备上进行推理任务。

关键词: 边缘智能, 设备端推理, 模型生成, 自动化部署, 边缘设备

Yusong TAN, Tian LI, Yusen ZHANG. Research on Neural Network Model Generation and Deployment for Edge Intelligence[J]. Computer Engineering, 2024, 50(8): 1-12.

谭郁松, 李恬, 张钰森. 面向边缘智能的神经网络模型生成与部署研究[J]. 计算机工程, 2024, 50(8): 1-12.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0068554

https://www.ecice06.com/EN/Y2024/V50/I8/1

Figures/Tables 14

Fig.1 Overall architecture of NGD system

Fig.2 Decoding rules of gradient optimization sampling

Fig.3 Transformation of the candidate neural network model

Fig.4 Performance evaluation process of the neural network model

Fig.5 Hierarchical storage structure

Fig.6 Overall structure of the retrieval interface

Fig.7 Result feedback strategy

Fig.8 Prototype system

Fig.9 Comparison of inference accuracy of the models before and after conversion

Fig.10 Comparison of inference latency of neural network models on different hardware platforms

Fig.11 Comparison of latency and accuracy of neural network models on different hardware platforms

References 28

1	中国国务院. 新一代人工智能发展规划. 科技导报, 2018, 36(17): 113. URL
	State Council of China. New generation artificial intelligence development plan. Science and Technology Herald, 2018, 36(17): 113. URL
2	杨鑫, 时晓厚, 沈云, 等. 5G工业互联网的边缘计算技术架构与应用. 电子技术应用, 2019, 45(12): 25-28, 33. URL
	YANG X, SHI X H, SHEN Y, et al. Edge computing applications and technical architecture of 5G industrial Internet. Application of Electronic Technique, 2019, 45(12): 25-28, 33. URL
3	冯怡, 李鑫, 江奎, 等. 物联网发展逻辑与商业模式创新. 通信企业管理, 2020,(5): 36- 40. URL
	FENG Y, LI X, JIANG K, et al. Development logic of Internet of Things and innovation of business model. C-Enterprise Management, 2020,(5): 36- 40. URL
4	DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional Transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Philadelphia, USA: Association for Computational Linguistics, 2019: 4176-4186.
5	高志强, 鲁晓阳, 张荣荣. 边缘智能: 关键技术与落地实践. 自动化博览, 2021, 38(10): 7. URL
	GAO Z Q, LU X Y, ZHANG R R. Edge intelligence: key technologies and landing practice. Automation Panorama, 2021, 38(10): 7. URL
6	牛鑫, 吕现伟, 余辰. 边缘智能: 现状与挑战. 武汉大学学报(理学版), 2023, 69(2): 270- 282. URL
	NIU X, LÜ X W, YU C. Edge intelligence: state-of-the-art and challenges. Journal of Wuhan University (Natural Science Edition), 2023, 69(2): 270- 282. URL
7	洪洲. 卷积神经网络的结构搜索方法研究[D]. 无锡: 江南大学, 2021.
	HONG Z. Research on convolutional neural network architecture search[D]. Wuxi: Jiangnan University, 2021. (in Chinese)
8	IANDOLA F N, HAN S, MOSKEWICZ M W, et al. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and < 0.5 MB model size[EB/OL]. [2023-09-10]. https://arxiv.org/abs/1602.07360.
9	HOWARD A G, ZHU M L, CHEN B, et al. MobileNets: efficient convolutional neural networks for mobile vision applications[EB/OL]. [2023-09-10]. https://arxiv.org/abs/1704.04861.
10	ZHANG X Y, ZHOU X Y, LIN M X, et al. ShuffleNet: an extremely efficient convolutional neural network for mobile devices[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 6848-6856.
11	葛道辉, 李洪升, 张亮, 等. 轻量级神经网络架构综述. 软件学报, 2020, 31(9): 2627- 2653. URL
	GE D H, LI H S, ZHANG L, et al. Survey of lightweight neural network. Journal of Software, 2020, 31(9): 2627- 2653. URL
12	罗人千. 高效神经网络结构搜索算法及应用[D]. 合肥: 中国科学技术大学, 2021.
	LUO R Q. Efficient neural architecture search: algorithms and applications[D]. Hefei: University of Science and Technology of China, 2021. (in Chinese)
13	YING C, KLEIN A, REAL E, et al. NAS-Bench-101: towards reproducible neural architecture search[C]//Proceedings of the International Conference on Machine Learning (ICML). Long Beach, USA: PMLR, 2019: 7105-7114.
14	DONG X Y, YANG Y. NAS-Bench-201: extending the scope of reproducible neural architecture search[EB/OL]. [2023-09-10]. https://arxiv.org/abs/2001.00326.
15	LI C J, YU Z Z, FU Y G, et al. HW-NAS-Bench: hardware-aware neural architecture search benchmark[EB/OL]. [2023-09-10]. https://arxiv.org/abs/2103.10584.
16	ZOPH B, VASUDEVAN V, SHLENS J, et al. Learning transferable architectures for scalable image recognition[C]//Proceedings of 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D. C., USA: IEEE Press, 2018: 8697-8710.
17	LIU H X, SIMONYAN K, YANG Y M. DARTS: differentiable architecture search[EB/OL]. [2023-09-10]. https://arxiv.org/abs/1806.09055.
18	XU Y, XIE L, ZHANG X, et al. PC-DARTS: partial channel connections for memory-efficient architecture search[EB/OL]. [2023-09-10]. https://arxiv.org/abs/1907.05737.
19	LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998, 86(11): 2278- 2324. doi: 10.1109/5.726791
20	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D. C., USA: IEEE Press, 2016: 770-778.
21	WAN A, DAI X L, ZHANG P Z, et al. FBNetV2: differentiable neural architecture search for spatial and channel dimensions[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D. C., USA: IEEE Press, 2020: 12965-12974.
22	清华大学. 轻量级神经网络模型训练方法、系统、装置及存储介质: CN202111080711.0[P]. 2021-12-17.
	Tsinghua University. Lightweight neural networks model training method, system, device and storage medium: CN202111080711.0[P]. 2021-12-17. (in Chinese)
23	卫佩宏. 边缘云中基于深度强化学习的任务调度及虚拟网络功能部署研究[D]. 重庆: 重庆理工大学, 2022.
	WEI P H. Task scheduling and virtual network function placement based on deep reinforcement learning in edge cloud[D]. Chongqing: Chongqing University of Technology, 2022. (in Chinese)
24	LU H D, DU M, HE X M, et al. An adaptive neural architecture search design for collaborative edge-cloud computing. IEEE Network, 2021, 35(5): 83- 89. doi: 10.1109/MNET.201.2100069
25	梅雅鑫. 阿里云: 打造三层边缘计算能力构建云边端协同的开放生态. 通信世界, 2019,(11): 44. URL
	MEI Y X. Alibaba cloud: building a three-tier edge computing capability and building an open ecology of cloud edge collaboration. Communications World, 2019,(11): 44. URL
26	MICHEL G, NIKOLENTZOS G, LUTZEYER J F, et al. Path neural networks: expressive and accurate graph neural networks[C]//Proceedings of International Conference on Machine Learning(ICML). Honolulu, USA: PMLR, 2023: 24737-24755.
27	WANG Y L, SU H, ZHANG B, et al. Interpret neural networks by identifying critical data routing paths[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 8906-8914.
28	YU F X, QIN Z W, CHEN X. Distilling critical paths in convolutional neural networks[EB/OL]. [2023-09-10]. https://arxiv.org/abs/1811.02643.

[1]	LI Zhijin, TANG Jiahui, YAN Jinfeng. Lightweight Recognition Method Based on Edge Computing [J]. Computer Engineering, 2024, 50(6): 287-295.
[2]	ZHANG Hao,LIU Yuan,WANG Xiaofeng,JIANG Min. Research on High-fidelity Link Emulation Based on Cloud Platform [J]. Computer Engineering, 2017, 43(8): 1-7.
[3]	WEI De-Zhi, HONG Lian-Ji, LIN Li-Na, WANG Ai-Guang. AMG Model Based on HGA and Data Mining [J]. Computer Engineering, 2012, 38(7): 99-101.

Please choose a citation manager

Content to export