一种面向智联网的高效联邦学习算法

doi:10.19678/j.issn.1000-3428.0066803

摘要/Abstract

摘要：

在智联网(AIoT)中引入联邦学习(FL)可以加强数据的隐私保护，然而分布式AIoT设备间的数据通常是非独立同分布的，标准的FL模型训练算法会使模型训练时出现客户机漂移的现象，导致收敛缓慢和不稳定。针对此问题，提出基于全局动量的联邦学习算法FedCNM。FedCNM将在AIoT服务器聚合的全局梯度信息发送至AIoT设备，让AIoT设备可以根据全局梯度信息来初始化本地模型，并标准化客户机模型的参数更新，以全局动量的方式平滑客户机模型的更新来缓解客户机漂移问题，加快模型的训练。在CIFAR-10和CIFAR-100数据集上模拟大规模设备、部分参与和不同数据分布场景进行仿真实验，结果表明，较对比方法，FedCNM在各种任务上训练的模型可以提高1.46%~11.12%的测试精度，且完成各种学习任务所需要的通信量最少。在CIFAR-10数据集上对比SGD+M、NAG、Adam和AMSGrad这4个本地优化器对算法的影响，实验结果表明，当本地使用基于动量的优化器SGD+M和NAG时，分别提高了10.53%和10.44%的测试精度。

关键词: 联邦学习, 动量, 智联网, 非独立同分布, 深度学习

Abstract:

Introducing Federated Learning(FL) in Artificial Intelligence of Things(AIoT) can enhance data privacy protection. However, data between distributed AIoT devices is usually non-Independent Identically Distribution(non-IID), and standard FL model training algorithms can cause client drift during model training, leading to slow convergence and instability. To address this issue, a FL algorithm FedCNM based on global momentum is proposed. FedCNM sends the global gradient information aggregated on the AIoT server to the AIoT device, allowing the AIoT device to initialize the local model based on the global gradient information and standardize the parameter updates of the client model, smoothing the updates of the client model in a global momentum manner to alleviate the problem of client drift and accelerate model training. Simulation experiments on large-scale devices, partial participation, and different data distributions were conducted on the CIFAR-10 and CIFAR-100 datasets. The results showed that the models trained by FedCNM on various tasks can improve testing accuracy by 1.46%-1.12%, and the communication required to complete various learning tasks is less than that of the comparison method. Comparing the impact of four local optimizers, SGD+M, NAG, Adam, and AMSGrad, on the CIFAR-10 dataset, the experimental results showed that when using momentum based optimizers SGD+M and NAG locally, the testing accuracy was improved by 10.53% and 10.44%, respectively.

Key words: Federated Learning(FL), momentum, Artificial Intelligence of Things(AIoT), non-Independent Identically Distribution(non-IID), deep learning

叶进, 韦涛, 胡亮青, 罗森, 李晓欢. 一种面向智联网的高效联邦学习算法[J]. 计算机工程, 2023, 49(12): 243-251, 261.

Jin YE, Tao WEI, Liangqing HU, Sen LUO, Xiaohuan LI. An Efficient Federated Learning Algorithm for Artificial Intelligence of Things[J]. Computer Engineering, 2023, 49(12): 243-251, 261.

https://www.ecice06.com/CN/Y2023/V49/I12/243

图/表 11

图1 基于全局动量的FL-AIoT结构模型

Fig.1 FL-AIoT structure model based on global momentum

图2 FedCNM的本地学习过程

Fig.2 Local learning process of FedCNM

图3 FedCNM与对比方法在中等规模设备下的收敛曲线图

Fig.3 Convergence curves of FedCNM and comparison methods in a moderate number of clients

图4 FedCNM与对比方法在大规模设备下的收敛曲线图

Fig.4 Convergence curves of FedCNM and comparison methods in a large number of clients

图5 FedCNM与对比方法在不同优化器下的收敛曲线图

Fig.5 Convergence curves of FedCNM and comparison methods under different optimizers

参考文献 34

1	VARGHESE J, VARGHEESE S K, PETER E. A study on artificial intelligence of things: techniques and applications. A Journal of Composition Theory, 2020, 8(3): 888- 896.
2	MCMAHAN H B, MOORE E, RAMAGE D, et al. Communication-efficient learning of deep networks from decentralized data[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1602.05629.pdf.
3	LI T, SAHU A K, TALWALKAR A, et al. Federated learning: challenges, methods, and future directions. IEEE Signal Processing Magazine, 2020, 37(3): 50- 60. doi: 10.1109/MSP.2020.2975749
4	KAIROUZ P, MCMAHAN H B, AVENT B, et al. Advances and open problems in federated learning. Foundations and Trends in Machine Learning, 2021, 14(1/2): 1- 210.
5	ZHAO Y, LI M, LAI L Z, et al. Federated learning with non-IID data[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1806.00582.pdf.
6	KARIMIREDDY S P, KALE S, MOHRI M, et al. SCAFFOLD: stochastic controlled averaging for federated learning[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1910.06378.pdf.
7	LI T, SAHU A K, ZAHEER M, et al. Federated optimization in heterogeneous networks[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1812.06127.pdf.
8	ACAR D A E, ZHAO Y, NAVARRO R M, et al. Federated learning based on dynamic regularization[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2111.04263.pdf.
9	WANG J Y, LIU Q H, LIANG H, et al. Tackling the objective inconsistency problem in heterogeneous federated optimization[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2007.07481.pdf.
10	REDDI S, CHARLES Z, ZAHEER M, et al. Adaptive federated optimization[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2003.00295.pdf.
11	ZHANG X W, HONG M Y, DHOPLE S, et al. FedPD: a federated learning framework with optimal rates and adaptivity to non-IID data[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2005.11418.pdf.
12	LIANG X F, SHEN S H, LIU J C, et al. Variance reduced local SGD with lower communication complexity[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1912.12844.pdf.
13	柏财通, 崔翛龙, 李爱. 基于本地蒸馏联邦学习的鲁棒语音识别技术. 计算机工程, 2022, 48(10): 103- 109. URL
	BAI C T, CUI X L, LI A. Robust speech recognition technology based on federal learning with local distillation. Computer Engineering, 2022, 48(10): 103- 109. URL
14	王树芬, 张哲, 马士尧, 等. 一种鲁棒的半监督联邦学习系统. 计算机工程, 2022, 48(6): 107-114, 123. URL
	WANG S F, ZHANG Z, MA S Y, et al. A robust semi-supervised federated learning system. Computer Engineering, 2022, 48(6): 107-114, 123. URL
15	SUTSKEVER I, MARTENS J, DAHL G, et al. On the importance of initialization and momentum in deep learning[C]//Proceedings of the 30th International Conference on Machine Learning. New York, USA: ACM Press, 2013: 1-9.
16	KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks. Communications of the ACM, 2017, 60(6): 84- 90. doi: 10.1145/3065386
17	OZFATURA E, OZFATURA K, GUNDUZ D. FedADC: accelerated federated learning with drift control[C]//Proceedings of 2021 IEEE International Symposium on Information Theory. Washington D. C., USA: IEEE Press, 2021: 467-472.
18	YANG Z J, BAO W, YUAN D, et al. Federated learning with Nesterov accelerated gradient. IEEE Transactions on Parallel and Distributed Systems, 2022, 33(12): 4863- 4873. doi: 10.1109/TPDS.2022.3206480
19	LIU W, CHEN L, CHEN Y F, et al. Accelerating federated learning via momentum gradient descent. IEEE Transactions on Parallel and Distributed Systems, 2020, 31(8): 1754- 1766. doi: 10.1109/TPDS.2020.2975189
20	HSU T M H, QI H, BROWN M. Measuring the effects of non-identical data distribution for federated visual classification[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1909.06335.pdf.
21	HUO Z Y, YANG Q, GU B, et al. Faster on-device training using new federated momentum algorithm[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2002.02090.pdf.
22	WANG J Y, TANTIA V, BALLAS N, et al. SlowMo: improving communication-efficient distributed SGD with slow momentum[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1910.00643.pdf.
23	KHANDURI P, SHARMA P, YANG H, et al. STEM: a stochastic two-sided momentum algorithm achieving near-optimal sample and communication complexities for federated learning. Advances in Neural Information Processing Systems, 2021, 34(1): 6050- 6061.
24	XU J, WANG S, WANG L W, et al. FedCM: federated learning with client-level momentum[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2106.10874.pdf.
25	XU A, HUANG H. Coordinating momenta for cross-silo federated learning. Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36(8): 8735- 8743. doi: 10.1609/aaai.v36i8.20853
26	DEVOLDER O, GLINEUR F, NESTEROV Y. First-order methods of smooth convex optimization with inexact oracle. Mathematical Programming, 2014, 146(1): 37- 75.
27	LESSARD L, RECHT B, PACKARD A. Analysis and design of optimization algorithms via integral quadratic constraints. SIAM Journal on Optimization, 2016, 26(1): 57- 95. doi: 10.1137/15M1009597
28	POLYAK B T. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, 1964, 4(5): 1- 17.
29	RUDER S. An overview of gradient descent optimization algorithms[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1609.04747.pdf.
30	WANG S Q, TUOR T, SALONIDIS T, et al. Adaptive federated learning in resource constrained edge computing systems. IEEE Journal on Selected Areas in Communications, 2019, 37(6): 1205- 1221.
31	KIM G, KIM J, HAN B. Communication-efficient federated learning with acceleration of global momentum[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2201.03172.pdf.
32	KRIZHEVSKY A, HINTON G. Learning multiple layers of features from tiny images[EB/OL]. [2023-01-02]. https://www.researchgate.net/publication/306218037_Learning_multiple_layers_of_features_from_tiny_images.
33	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 770-778.
34	NESTEROV Y. A method for unconstrained convex minimization problem with the rate of convergence O(1/k^2). Doklady Akademii Nauk SSSR, 1983, 269(3): 543- 547.

[1]	魏嵬, 丁香香, 郭梦星, 杨钊, 刘辉. 文本相似度计算方法综述[J]. 计算机工程, 2024, 50(9): 18-32.
[2]	朱凯, 李理, 张彤, 江晟, 别一鸣. 基于Transformer的多阶段运动模糊图像修复网络[J]. 计算机工程, 2024, 50(9): 276-285.
[3]	张天鹏, 韩晶, 吕学强. 基于多任务学习的超分辨率辅助小目标检测[J]. 计算机工程, 2024, 50(9): 304-312.
[4]	高煜宝, 文志诚. 基于注意力机制的双路解码器图像去噪方法[J]. 计算机工程, 2024, 50(9): 324-332.
[5]	潘恩元, 钟原, 李平. 联邦异质性数据下半监督颈椎MRI分割模型[J]. 计算机工程, 2024, 50(9): 367-376.
[6]	张华青, 夏张涛, 陆晓庆, 童基均. 基于字形特征的血管外科命名实体识别[J]. 计算机工程, 2024, 50(8): 13-21.
[7]	张亚洲, 和玉, 戎璐, 王祥凯. 基于上下文知识增强型Transformer网络的抑郁检测[J]. 计算机工程, 2024, 50(8): 75-85.
[8]	李红娇, 王宝金, 王朝晖, 胡仁豪. 基于模型相似度与本地损失的双重客户端选择算法[J]. 计算机工程, 2024, 50(8): 153-164.
[9]	高伟, 李帅龙, 茆琳, 王磊, 李颖颖, 韩林. 一种基于TVM的算子生成加速策略[J]. 计算机工程, 2024, 50(8): 353-362.
[10]	王宇, 祁琦, 王纯, 许才. 储能变流器信号高精度故障诊断方法[J]. 计算机工程, 2024, 50(8): 389-396.
[11]	牛瑞婷, 严天峰, 高锐, 王映植. 低信噪比下基于深度学习TCNN-MobileNet的调制识别[J]. 计算机工程, 2024, 50(7): 204-215.
[12]	肖慈, 徐杨, 张永丹, 冯明文, 黄易仟. 结合注意力和低光增强的夜间语义分割[J]. 计算机工程, 2024, 50(7): 271-281.
[13]	张诗婧, 莫绪涛, 赵行, 董杨林. 基于球面折反射成像和YOLOv7的内螺纹缺陷检测[J]. 计算机工程, 2024, 50(7): 282-292.
[14]	徐明亮, 李芳媛, 马浩然, 何飞. 大规模神经记录的峰电位聚类算法(特邀)[J]. 计算机工程, 2024, 50(6): 1-34.
[15]	魏琢艺, 罗迈, 李文兵, 曾远松, 余伟江, 杨跃东. 基于多源域适应的单细胞智能分类[J]. 计算机工程, 2024, 50(6): 48-55.

选择文件类型/文献管理软件名称

选择包含的内容