一种面向智联网的高效联邦学习算法

doi:10.19678/j.issn.1000-3428.0066803

摘要/Abstract

摘要：

在智联网(AIoT)中引入联邦学习(FL)可以加强数据的隐私保护，然而分布式AIoT设备间的数据通常是非独立同分布的，标准的FL模型训练算法会使模型训练时出现客户机漂移的现象，导致收敛缓慢和不稳定。针对此问题，提出基于全局动量的联邦学习算法FedCNM。FedCNM将在AIoT服务器聚合的全局梯度信息发送至AIoT设备，让AIoT设备可以根据全局梯度信息来初始化本地模型，并标准化客户机模型的参数更新，以全局动量的方式平滑客户机模型的更新来缓解客户机漂移问题，加快模型的训练。在CIFAR-10和CIFAR-100数据集上模拟大规模设备、部分参与和不同数据分布场景进行仿真实验，结果表明，较对比方法，FedCNM在各种任务上训练的模型可以提高1.46%~11.12%的测试精度，且完成各种学习任务所需要的通信量最少。在CIFAR-10数据集上对比SGD+M、NAG、Adam和AMSGrad这4个本地优化器对算法的影响，实验结果表明，当本地使用基于动量的优化器SGD+M和NAG时，分别提高了10.53%和10.44%的测试精度。

关键词: 联邦学习, 动量, 智联网, 非独立同分布, 深度学习

Abstract:

Introducing Federated Learning(FL) in Artificial Intelligence of Things(AIoT) can enhance data privacy protection. However, data between distributed AIoT devices is usually non-Independent Identically Distribution(non-IID), and standard FL model training algorithms can cause client drift during model training, leading to slow convergence and instability. To address this issue, a FL algorithm FedCNM based on global momentum is proposed. FedCNM sends the global gradient information aggregated on the AIoT server to the AIoT device, allowing the AIoT device to initialize the local model based on the global gradient information and standardize the parameter updates of the client model, smoothing the updates of the client model in a global momentum manner to alleviate the problem of client drift and accelerate model training. Simulation experiments on large-scale devices, partial participation, and different data distributions were conducted on the CIFAR-10 and CIFAR-100 datasets. The results showed that the models trained by FedCNM on various tasks can improve testing accuracy by 1.46%-1.12%, and the communication required to complete various learning tasks is less than that of the comparison method. Comparing the impact of four local optimizers, SGD+M, NAG, Adam, and AMSGrad, on the CIFAR-10 dataset, the experimental results showed that when using momentum based optimizers SGD+M and NAG locally, the testing accuracy was improved by 10.53% and 10.44%, respectively.

Key words: Federated Learning(FL), momentum, Artificial Intelligence of Things(AIoT), non-Independent Identically Distribution(non-IID), deep learning

叶进, 韦涛, 胡亮青, 罗森, 李晓欢. 一种面向智联网的高效联邦学习算法[J]. 计算机工程, 2023, 49(12): 243-251, 261.

Jin YE, Tao WEI, Liangqing HU, Sen LUO, Xiaohuan LI. An Efficient Federated Learning Algorithm for Artificial Intelligence of Things[J]. Computer Engineering, 2023, 49(12): 243-251, 261.

http://www.ecice06.com/CN/Y2023/V49/I12/243

图/表 11

图1 基于全局动量的FL-AIoT结构模型

Fig.1 FL-AIoT structure model based on global momentum

图2 FedCNM的本地学习过程

Fig.2 Local learning process of FedCNM

图3 FedCNM与对比方法在中等规模设备下的收敛曲线图

Fig.3 Convergence curves of FedCNM and comparison methods in a moderate number of clients

图4 FedCNM与对比方法在大规模设备下的收敛曲线图

Fig.4 Convergence curves of FedCNM and comparison methods in a large number of clients

图5 FedCNM与对比方法在不同优化器下的收敛曲线图

Fig.5 Convergence curves of FedCNM and comparison methods under different optimizers

参考文献 34

1	VARGHESE J, VARGHEESE S K, PETER E. A study on artificial intelligence of things: techniques and applications. A Journal of Composition Theory, 2020, 8(3): 888- 896.
2	MCMAHAN H B, MOORE E, RAMAGE D, et al. Communication-efficient learning of deep networks from decentralized data[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1602.05629.pdf.
3	LI T, SAHU A K, TALWALKAR A, et al. Federated learning: challenges, methods, and future directions. IEEE Signal Processing Magazine, 2020, 37(3): 50- 60. doi: 10.1109/MSP.2020.2975749
4	KAIROUZ P, MCMAHAN H B, AVENT B, et al. Advances and open problems in federated learning. Foundations and Trends in Machine Learning, 2021, 14(1/2): 1- 210.
5	ZHAO Y, LI M, LAI L Z, et al. Federated learning with non-IID data[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1806.00582.pdf.
6	KARIMIREDDY S P, KALE S, MOHRI M, et al. SCAFFOLD: stochastic controlled averaging for federated learning[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1910.06378.pdf.
7	LI T, SAHU A K, ZAHEER M, et al. Federated optimization in heterogeneous networks[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1812.06127.pdf.
8	ACAR D A E, ZHAO Y, NAVARRO R M, et al. Federated learning based on dynamic regularization[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2111.04263.pdf.
9	WANG J Y, LIU Q H, LIANG H, et al. Tackling the objective inconsistency problem in heterogeneous federated optimization[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2007.07481.pdf.
10	REDDI S, CHARLES Z, ZAHEER M, et al. Adaptive federated optimization[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2003.00295.pdf.
11	ZHANG X W, HONG M Y, DHOPLE S, et al. FedPD: a federated learning framework with optimal rates and adaptivity to non-IID data[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2005.11418.pdf.
12	LIANG X F, SHEN S H, LIU J C, et al. Variance reduced local SGD with lower communication complexity[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1912.12844.pdf.
13	柏财通, 崔翛龙, 李爱. 基于本地蒸馏联邦学习的鲁棒语音识别技术. 计算机工程, 2022, 48(10): 103- 109. URL
	BAI C T, CUI X L, LI A. Robust speech recognition technology based on federal learning with local distillation. Computer Engineering, 2022, 48(10): 103- 109. URL
14	王树芬, 张哲, 马士尧, 等. 一种鲁棒的半监督联邦学习系统. 计算机工程, 2022, 48(6): 107-114, 123. URL
	WANG S F, ZHANG Z, MA S Y, et al. A robust semi-supervised federated learning system. Computer Engineering, 2022, 48(6): 107-114, 123. URL
15	SUTSKEVER I, MARTENS J, DAHL G, et al. On the importance of initialization and momentum in deep learning[C]//Proceedings of the 30th International Conference on Machine Learning. New York, USA: ACM Press, 2013: 1-9.
16	KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks. Communications of the ACM, 2017, 60(6): 84- 90. doi: 10.1145/3065386
17	OZFATURA E, OZFATURA K, GUNDUZ D. FedADC: accelerated federated learning with drift control[C]//Proceedings of 2021 IEEE International Symposium on Information Theory. Washington D. C., USA: IEEE Press, 2021: 467-472.
18	YANG Z J, BAO W, YUAN D, et al. Federated learning with Nesterov accelerated gradient. IEEE Transactions on Parallel and Distributed Systems, 2022, 33(12): 4863- 4873. doi: 10.1109/TPDS.2022.3206480
19	LIU W, CHEN L, CHEN Y F, et al. Accelerating federated learning via momentum gradient descent. IEEE Transactions on Parallel and Distributed Systems, 2020, 31(8): 1754- 1766. doi: 10.1109/TPDS.2020.2975189
20	HSU T M H, QI H, BROWN M. Measuring the effects of non-identical data distribution for federated visual classification[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1909.06335.pdf.
21	HUO Z Y, YANG Q, GU B, et al. Faster on-device training using new federated momentum algorithm[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2002.02090.pdf.
22	WANG J Y, TANTIA V, BALLAS N, et al. SlowMo: improving communication-efficient distributed SGD with slow momentum[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1910.00643.pdf.
23	KHANDURI P, SHARMA P, YANG H, et al. STEM: a stochastic two-sided momentum algorithm achieving near-optimal sample and communication complexities for federated learning. Advances in Neural Information Processing Systems, 2021, 34(1): 6050- 6061.
24	XU J, WANG S, WANG L W, et al. FedCM: federated learning with client-level momentum[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2106.10874.pdf.
25	XU A, HUANG H. Coordinating momenta for cross-silo federated learning. Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36(8): 8735- 8743. doi: 10.1609/aaai.v36i8.20853
26	DEVOLDER O, GLINEUR F, NESTEROV Y. First-order methods of smooth convex optimization with inexact oracle. Mathematical Programming, 2014, 146(1): 37- 75.
27	LESSARD L, RECHT B, PACKARD A. Analysis and design of optimization algorithms via integral quadratic constraints. SIAM Journal on Optimization, 2016, 26(1): 57- 95. doi: 10.1137/15M1009597
28	POLYAK B T. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, 1964, 4(5): 1- 17.
29	RUDER S. An overview of gradient descent optimization algorithms[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1609.04747.pdf.
30	WANG S Q, TUOR T, SALONIDIS T, et al. Adaptive federated learning in resource constrained edge computing systems. IEEE Journal on Selected Areas in Communications, 2019, 37(6): 1205- 1221.
31	KIM G, KIM J, HAN B. Communication-efficient federated learning with acceleration of global momentum[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2201.03172.pdf.
32	KRIZHEVSKY A, HINTON G. Learning multiple layers of features from tiny images[EB/OL]. [2023-01-02]. https://www.researchgate.net/publication/306218037_Learning_multiple_layers_of_features_from_tiny_images.
33	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 770-778.
34	NESTEROV Y. A method for unconstrained convex minimization problem with the rate of convergence O(1/k^2). Doklady Akademii Nauk SSSR, 1983, 269(3): 543- 547.

[1]	池亚平, 岳梓岩, 林雨衡. 基于Transformer的SM4算法工作模式识别[J]. 计算机工程, 2023, 49(9): 109-117.
[2]	林中霖, 时金桥, 王美琪, 王学宾, 王雨燕. 基于应用行为划分的Android恶意应用检测技术[J]. 计算机工程, 2023, 49(9): 125-136.
[3]	张攀峰, 吴丹华, 董明刚. 基于粒子群优化的差分隐私深度学习模型[J]. 计算机工程, 2023, 49(9): 144-157.
[4]	江雨燕, 陶承凤, 李平. 数据增强和自适应自步学习的深度子空间聚类算法[J]. 计算机工程, 2023, 49(8): 96-103, 110.
[5]	郑美光, 杨泳. 基于互信息软聚类的个性化联邦学习算法[J]. 计算机工程, 2023, 49(8): 20-28.
[6]	李泽水, 冀俊忠, 杨翠翠. 基于边权重信息深度网络嵌入的PPIN功能模块检测[J]. 计算机工程, 2023, 49(8): 69-76.
[7]	王可铮, 徐玉芬, 周尚波. 结合对比感知损失和融合注意力的图像去雾模型[J]. 计算机工程, 2023, 49(8): 207-214.
[8]	刘俊豪, 王美林, 谢兴, 宋烨兴, 许莉花. 基于改进YOLOv5的皮革瑕疵检测算法[J]. 计算机工程, 2023, 49(8): 240-249.
[9]	邱天晨, 郑小盈, 祝永新, 封松林. 面向非独立同分布数据的联邦学习架构[J]. 计算机工程, 2023, 49(7): 110-117.
[10]	闫兴亚, 匡娅茜, 白光睿, 李月. 基于深度学习的学生课堂行为识别方法[J]. 计算机工程, 2023, 49(7): 251-258.
[11]	李军侠, 王星驰, 殷梓, 石德硕. 边缘深度挖掘的弱监督显著性目标检测[J]. 计算机工程, 2023, 49(7): 169-178.
[12]	吴珊, 周凤. 基于改进SSD算法的小目标检测[J]. 计算机工程, 2023, 49(7): 179-188.
[13]	席建锐, 唐红梅, 梁春阳, 刘鑫. 基于改进隐函数的点云物体重建[J]. 计算机工程, 2023, 49(7): 214-222.
[14]	齐咏生, 杜晓旭, 朱俊峰, 高胜利, 刘利强. 基于增强型轻量深度网络的牧区牲畜高效检测[J]. 计算机工程, 2023, 49(7): 278-287.
[15]	谌雨章, 黄逸姿, 张钧涵. 基于多速率空洞卷积的多尺度水下小目标检测[J]. 计算机工程, 2023, 49(6): 257-264.

选择文件类型/文献管理软件名称

选择包含的内容