An Efficient Federated Learning Algorithm for Artificial Intelligence of Things

doi:10.19678/j.issn.1000-3428.0066803

Abstract

Abstract:

Introducing Federated Learning(FL) in Artificial Intelligence of Things(AIoT) can enhance data privacy protection. However, data between distributed AIoT devices is usually non-Independent Identically Distribution(non-IID), and standard FL model training algorithms can cause client drift during model training, leading to slow convergence and instability. To address this issue, a FL algorithm FedCNM based on global momentum is proposed. FedCNM sends the global gradient information aggregated on the AIoT server to the AIoT device, allowing the AIoT device to initialize the local model based on the global gradient information and standardize the parameter updates of the client model, smoothing the updates of the client model in a global momentum manner to alleviate the problem of client drift and accelerate model training. Simulation experiments on large-scale devices, partial participation, and different data distributions were conducted on the CIFAR-10 and CIFAR-100 datasets. The results showed that the models trained by FedCNM on various tasks can improve testing accuracy by 1.46%-1.12%, and the communication required to complete various learning tasks is less than that of the comparison method. Comparing the impact of four local optimizers, SGD+M, NAG, Adam, and AMSGrad, on the CIFAR-10 dataset, the experimental results showed that when using momentum based optimizers SGD+M and NAG locally, the testing accuracy was improved by 10.53% and 10.44%, respectively.

Key words: Federated Learning(FL), momentum, Artificial Intelligence of Things(AIoT), non-Independent Identically Distribution(non-IID), deep learning

摘要：

在智联网(AIoT)中引入联邦学习(FL)可以加强数据的隐私保护，然而分布式AIoT设备间的数据通常是非独立同分布的，标准的FL模型训练算法会使模型训练时出现客户机漂移的现象，导致收敛缓慢和不稳定。针对此问题，提出基于全局动量的联邦学习算法FedCNM。FedCNM将在AIoT服务器聚合的全局梯度信息发送至AIoT设备，让AIoT设备可以根据全局梯度信息来初始化本地模型，并标准化客户机模型的参数更新，以全局动量的方式平滑客户机模型的更新来缓解客户机漂移问题，加快模型的训练。在CIFAR-10和CIFAR-100数据集上模拟大规模设备、部分参与和不同数据分布场景进行仿真实验，结果表明，较对比方法，FedCNM在各种任务上训练的模型可以提高1.46%~11.12%的测试精度，且完成各种学习任务所需要的通信量最少。在CIFAR-10数据集上对比SGD+M、NAG、Adam和AMSGrad这4个本地优化器对算法的影响，实验结果表明，当本地使用基于动量的优化器SGD+M和NAG时，分别提高了10.53%和10.44%的测试精度。

关键词: 联邦学习, 动量, 智联网, 非独立同分布, 深度学习

Jin YE, Tao WEI, Liangqing HU, Sen LUO, Xiaohuan LI. An Efficient Federated Learning Algorithm for Artificial Intelligence of Things[J]. Computer Engineering, 2023, 49(12): 243-251, 261.

叶进, 韦涛, 胡亮青, 罗森, 李晓欢. 一种面向智联网的高效联邦学习算法[J]. 计算机工程, 2023, 49(12): 243-251, 261.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0066803

https://www.ecice06.com/EN/Y2023/V49/I12/243

Figures/Tables 11

Fig.1 FL-AIoT structure model based on global momentum

Fig.2 Local learning process of FedCNM

Fig.3 Convergence curves of FedCNM and comparison methods in a moderate number of clients

Fig.4 Convergence curves of FedCNM and comparison methods in a large number of clients

Fig.5 Convergence curves of FedCNM and comparison methods under different optimizers

References 34

1	VARGHESE J, VARGHEESE S K, PETER E. A study on artificial intelligence of things: techniques and applications. A Journal of Composition Theory, 2020, 8(3): 888- 896.
2	MCMAHAN H B, MOORE E, RAMAGE D, et al. Communication-efficient learning of deep networks from decentralized data[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1602.05629.pdf.
3	LI T, SAHU A K, TALWALKAR A, et al. Federated learning: challenges, methods, and future directions. IEEE Signal Processing Magazine, 2020, 37(3): 50- 60. doi: 10.1109/MSP.2020.2975749
4	KAIROUZ P, MCMAHAN H B, AVENT B, et al. Advances and open problems in federated learning. Foundations and Trends in Machine Learning, 2021, 14(1/2): 1- 210.
5	ZHAO Y, LI M, LAI L Z, et al. Federated learning with non-IID data[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1806.00582.pdf.
6	KARIMIREDDY S P, KALE S, MOHRI M, et al. SCAFFOLD: stochastic controlled averaging for federated learning[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1910.06378.pdf.
7	LI T, SAHU A K, ZAHEER M, et al. Federated optimization in heterogeneous networks[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1812.06127.pdf.
8	ACAR D A E, ZHAO Y, NAVARRO R M, et al. Federated learning based on dynamic regularization[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2111.04263.pdf.
9	WANG J Y, LIU Q H, LIANG H, et al. Tackling the objective inconsistency problem in heterogeneous federated optimization[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2007.07481.pdf.
10	REDDI S, CHARLES Z, ZAHEER M, et al. Adaptive federated optimization[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2003.00295.pdf.
11	ZHANG X W, HONG M Y, DHOPLE S, et al. FedPD: a federated learning framework with optimal rates and adaptivity to non-IID data[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2005.11418.pdf.
12	LIANG X F, SHEN S H, LIU J C, et al. Variance reduced local SGD with lower communication complexity[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1912.12844.pdf.
13	柏财通, 崔翛龙, 李爱. 基于本地蒸馏联邦学习的鲁棒语音识别技术. 计算机工程, 2022, 48(10): 103- 109. URL
	BAI C T, CUI X L, LI A. Robust speech recognition technology based on federal learning with local distillation. Computer Engineering, 2022, 48(10): 103- 109. URL
14	王树芬, 张哲, 马士尧, 等. 一种鲁棒的半监督联邦学习系统. 计算机工程, 2022, 48(6): 107-114, 123. URL
	WANG S F, ZHANG Z, MA S Y, et al. A robust semi-supervised federated learning system. Computer Engineering, 2022, 48(6): 107-114, 123. URL
15	SUTSKEVER I, MARTENS J, DAHL G, et al. On the importance of initialization and momentum in deep learning[C]//Proceedings of the 30th International Conference on Machine Learning. New York, USA: ACM Press, 2013: 1-9.
16	KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks. Communications of the ACM, 2017, 60(6): 84- 90. doi: 10.1145/3065386
17	OZFATURA E, OZFATURA K, GUNDUZ D. FedADC: accelerated federated learning with drift control[C]//Proceedings of 2021 IEEE International Symposium on Information Theory. Washington D. C., USA: IEEE Press, 2021: 467-472.
18	YANG Z J, BAO W, YUAN D, et al. Federated learning with Nesterov accelerated gradient. IEEE Transactions on Parallel and Distributed Systems, 2022, 33(12): 4863- 4873. doi: 10.1109/TPDS.2022.3206480
19	LIU W, CHEN L, CHEN Y F, et al. Accelerating federated learning via momentum gradient descent. IEEE Transactions on Parallel and Distributed Systems, 2020, 31(8): 1754- 1766. doi: 10.1109/TPDS.2020.2975189
20	HSU T M H, QI H, BROWN M. Measuring the effects of non-identical data distribution for federated visual classification[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1909.06335.pdf.
21	HUO Z Y, YANG Q, GU B, et al. Faster on-device training using new federated momentum algorithm[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2002.02090.pdf.
22	WANG J Y, TANTIA V, BALLAS N, et al. SlowMo: improving communication-efficient distributed SGD with slow momentum[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1910.00643.pdf.
23	KHANDURI P, SHARMA P, YANG H, et al. STEM: a stochastic two-sided momentum algorithm achieving near-optimal sample and communication complexities for federated learning. Advances in Neural Information Processing Systems, 2021, 34(1): 6050- 6061.
24	XU J, WANG S, WANG L W, et al. FedCM: federated learning with client-level momentum[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2106.10874.pdf.
25	XU A, HUANG H. Coordinating momenta for cross-silo federated learning. Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36(8): 8735- 8743. doi: 10.1609/aaai.v36i8.20853
26	DEVOLDER O, GLINEUR F, NESTEROV Y. First-order methods of smooth convex optimization with inexact oracle. Mathematical Programming, 2014, 146(1): 37- 75.
27	LESSARD L, RECHT B, PACKARD A. Analysis and design of optimization algorithms via integral quadratic constraints. SIAM Journal on Optimization, 2016, 26(1): 57- 95. doi: 10.1137/15M1009597
28	POLYAK B T. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, 1964, 4(5): 1- 17.
29	RUDER S. An overview of gradient descent optimization algorithms[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1609.04747.pdf.
30	WANG S Q, TUOR T, SALONIDIS T, et al. Adaptive federated learning in resource constrained edge computing systems. IEEE Journal on Selected Areas in Communications, 2019, 37(6): 1205- 1221.
31	KIM G, KIM J, HAN B. Communication-efficient federated learning with acceleration of global momentum[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2201.03172.pdf.
32	KRIZHEVSKY A, HINTON G. Learning multiple layers of features from tiny images[EB/OL]. [2023-01-02]. https://www.researchgate.net/publication/306218037_Learning_multiple_layers_of_features_from_tiny_images.
33	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 770-778.
34	NESTEROV Y. A method for unconstrained convex minimization problem with the rate of convergence O(1/k^2). Doklady Akademii Nauk SSSR, 1983, 269(3): 543- 547.

[1]	WEI Wei, DING Xiangxiang, GUO Mengxing, YANG Zhao, LIU Hui. Review of Text Similarity Calculation Methods [J]. Computer Engineering, 2024, 50(9): 18-32.
[2]	ZHANG Tianpeng, HAN Jing, LÜ Xueqiang. Super-Resolution-Aided Small-Target Detection Based on Multi-Task Learning [J]. Computer Engineering, 2024, 50(9): 304-312.
[3]	GAO Yubao, WEN Zhicheng. Dual Decoder Image Denoising Method Based on Attention Mechanism [J]. Computer Engineering, 2024, 50(9): 324-332.
[4]	ZHU Kai, LI Li, ZHANG Tong, JIANG Sheng, BIE Yiming. Multi-Stage Motion Blur Image Restoration Network Based on Transformer [J]. Computer Engineering, 2024, 50(9): 276-285.
[5]	Huaqing ZHANG, Zhangtao XIA, Xiaoqing LU, Jijun TONG. Named Entity Recognition of Vascular Surgery Based on Glyph Features [J]. Computer Engineering, 2024, 50(8): 13-21.
[6]	Yazhou ZHANG, Yu HE, Lu RONG, Xiangkai WANG. Depression Detection Based on Contextual Knowledge Enhanced Transformer Network [J]. Computer Engineering, 2024, 50(8): 75-85.
[7]	Wei GAO, Shuailong LI, Lin MAO, Lei WANG, Yingying LI, Lin HAN. One Acceleration Strategy for Operator Generation Based on TVM [J]. Computer Engineering, 2024, 50(8): 353-362.
[8]	Yu WANG, Qi QI, Chun WANG, Cai XU. High-Precision Fault Diagnosis Method for Energy Storage Inverter Signals [J]. Computer Engineering, 2024, 50(8): 389-396.
[9]	Ci XIAO, Yang XU, Yongdan ZHANG, Mingwen FENG, Yiqian HUANG. Nighttime Semantic Segmentation with Attention and Low-Light Enhancement [J]. Computer Engineering, 2024, 50(7): 271-281.
[10]	Shijing ZHANG, Xutao MO, Xing ZHAO, Yanglin DONG. Internal Thread Defect Detection Based on Spherical Catadioptric Imaging and YOLOv7 [J]. Computer Engineering, 2024, 50(7): 282-292.
[11]	Ruiting NUI, Tianfeng YAN, Rui GAO, Yingzhi WANG. Deep Learning TCNN-MobileNet-Based Modulation Recognition Under Low Signal-to-Noise Radio [J]. Computer Engineering, 2024, 50(7): 204-215.
[12]	XU Mingliang, LI Fangyuan, MA Haoran, HE Fei. Spike Sorting Algorithms for Large-Scale Neural Recording [J]. Computer Engineering, 2024, 50(6): 1-34.
[13]	WEI Zhuoyi, LUO Mai, LI Wenbing, ZENG Yuansong, YU Weijiang, YANG Yuedong. Intelligent Single-Cell Classification Based on Multisource Domain Adaptation [J]. Computer Engineering, 2024, 50(6): 48-55.
[14]	QIN Yuan, ZHANG Hang, ZHU Hongpeng, LI Jiong, HU Hang. Online BSS Algorithm for Anti-jamming of Mobile UAV Cluster in Satellite MIMO Communication System [J]. Computer Engineering, 2024, 50(6): 65-76.
[15]	LI Zijie, ZHOU Juxiang, HAN Xiaoyu, GAN Jianhou, LU Zeguang, WANG Jun. Knowledge Tracing Model Based on the Fusion of Sequence Features and Learning Processes [J]. Computer Engineering, 2024, 50(6): 77-85.

Please choose a citation manager

Content to export