非独立同分布场景下的联邦学习优化方法

doi:10.19678/j.issn.1000-3428.0067791

摘要/Abstract

摘要：

联邦学习能够在不泄露数据隐私的情况下合作训练全局模型，但这种协作式的训练方式在现实环境下面临参与方数据非独立同分布（Non-IID）的挑战：模型收敛慢、精度降低的问题。许多现有的联邦学习方法仅从全局模型聚合和本地客户端更新中的一个角度进行改进，难免会引发另一角度带来的影响，降低全局模型的质量。提出一种分层持续学习的联邦学习优化方法（FedMas）。FedMas基于分层融合的思想，首先，采用客户端分层策略，利用DBSCAN算法将相似数据分布的客户端划分到不同的层中，每次仅挑选某个层的部分客户端进行训练，避免服务器端全局模型聚合时因数据分布不同产生的权重分歧现象；进一步，由于每个层的数据分布不同，客户端在局部更新时结合持续学习灾难性遗忘的解决方案，有效地融合不同层客户端数据间的差异性，从而保证全局模型的性能。在MNIST和CIFAR-10标准数据集上的实验结果表明，FedMas与FedProx、Scaffold和FedCurv联邦学习算法相比，全局模型测试准确率平均提高0.3~2.2个百分点。

关键词: 联邦学习, 持续学习, 数据异构, 聚类, 分层优化, 数据分布

Abstract:

Federated Learning(FL) can collaborate to train global models without compromising data privacy. Nonetheless, this collaborative training approach faces the challenge of Non-IID in the real world; slow model convergence and low accuracy. Numerous existing FL methods improve only from one perspective of global model aggregation and local client update, and inevitably will not cause the impact of the other perspective and reduce the quality of the global model. In this context, we introduce a hierarchical continuous learning optimization method for FL, denoted as FedMas, which is based on the idea of hierarchical fusion. First, clients with similar data distribution are divided into different layers using the DBSCAN algorithm, and only part of clients of a certain layer are selected for training each time to avoid weight differences caused by different data distributions when the server global model is aggregated. Further, owing to the different data distributions of each layer, the client combines the solution of continuous learning catastrophic forgetting during local update to effectively integrate the differences between the data of different layers of clients, thus ensuring the performance of the global model. Experiments on MNIST and CIFAR-10 standard datasets demonstrate that the global model test accuracy is improved by 0.3-2.2 percentage points on average compared with FedProx, Scaffold, and FedCurv FL algorithms.

Key words: Federated Learning(FL), continual learning, data heterogeneity, clustering, hierarchical optimization, data distribution

宋华伟, 李升起, 万方杰, 卫玉萍. 非独立同分布场景下的联邦学习优化方法[J]. 计算机工程, 2024, 50(3): 166-172.

Huawei SONG, Shengqi LI, Fangjie WAN, Yuping WEI. Federated Learning Optimization Method in Non-IID Scenarios[J]. Computer Engineering, 2024, 50(3): 166-172.

http://www.ecice06.com/CN/Y2024/V50/I3/166

图/表 7

图1 Non-IID数据训练时的模型偏移

Fig.1 Model offset during Non-IID data training

图2 FedMas整体架构

Fig.2 Overall architecture of FedMas

图3 MNIST数据集在不同异构设置下前10个客户端的数据分布图

Fig.3 First ten clients' data distribution in MNIST dataset under different data heterogeneities

图4 分层和局部更新的有效性

Fig.4 Effectiveness of hierarchical and local updates

图5 α=0.3时FedCurv、FedProx、Scaffold和FedMas在MNIST数据集上的准确率

Fig.5 The accuracy of FedCurv, FedProx, Scaffold and FedMas on the MNIST dataset when α=0.3

图6 α=0.3时FedCurv、FedProx、Scaffold和FedMas在CIFAR-10数据集上下的准确率

Fig.6 The accuracy of FedCurv, FedProx, Scaffold and FedMas on the CIFAR-10 dataset when α=0.3

参考文献 27

1	HASAN M. State of IoT 2022: number of connected IoT devices growing 18% to 14.4 billion globally[EB/OL]. [2023-02-20]. https://iot-analytics.com/number-connected-iot-devices/.
2	QI Q S, XU Z Y, RANI P. Big data analytics challenges to implementing the intelligent Industrial Internet of Things(IIoT) systems in sustainable manufacturing operations. Technological Forecasting and Social Change, 2023, 190, 122401. doi: 10.1016/j.techfore.2023.122401
3	赵朝阳, 朱贵波, 王金桥. ChatGPT给语言大模型带来的启示和多模态大模型新的发展思路. 数据分析与知识发现, 2023, 7(3): 26- 35. URL
	ZHAO C Y, ZHU G B, WANG J Q. The inspiration brought by ChatGPT to LLM and the new development ideas of multi-modal large model. Data Analysis and Knowledge Discovery, 2023, 7(3): 26- 35. URL
4	GAFF B M, SUSSMAN H E, GEETTER J. Privacy and big data. Computer, 2014, 47(6): 7- 9. doi: 10.1109/MC.2014.161
5	李少波, 杨磊, 李传江, 等. 联邦学习概述: 技术、应用及未来. 计算机集成制造系统, 2022, 28(7): 2119- 2138. URL
	LI S B, YANG L, LI C J, et al. Overview of federated learning: technology, applications and future. Computer Integrated Manufacturing Systems, 2022, 28(7): 2119- 2138. URL
6	MCMAHAN H B, MOORE E, RAMAGE D, et al. Communication-efficient learning of deep networks from decentralized data[EB/OL]. [2023-02-20]. https://arxiv.org/abs/1602.05629.pdf.
7	SATTLER F, WIEDEMANN S, MULLER K R, et al. Robust and communication-efficient federated learning from non-i. i. d. data. IEEE Transactions on Neural Networks and Learning Systems, 2020, 31(9): 3400- 3413. doi: 10.1109/TNNLS.2019.2944481
8	BOTTOU L. Large-scale machine learning with stochastic gradient descent[C]//Proceedings of COMPSTAT'10. Heidelberg, Germany: Physica-Verlag HD, 2010: 177-186.
9	ZHAO Y, LI M, LAI L, et al. Federated learning with Non-IID data[EB/OL]. [2023-02-20]. https://arxiv.org/abs/1806.00582.
10	LI X, HUANG K, YANG W, et al. On the convergence of FedAvg on Non-IID data[EB/OL]. [2023-02-20]. https://arxiv.org/abs/1907.02189v2.
11	WANG J, LIU Q, LIANG H, et al. Tackling the objective inconsistency problem in heterogeneous federated optimization[EB/OL]. [2023-02-20]. https://arxiv.org/abs/2007.07481.
12	SATTLER F, MULLER K R, SAMEK W. Clustered federated learning: model-agnostic distributed multitask optimization under privacy constraints. IEEE Transactions on Neural Networks and Learning Systems, 2021, 32(8): 3710- 3722. doi: 10.1109/TNNLS.2020.3015958
13	HUANG W, LI T R, WANG D X, et al. Fair and accuracy in federated learning[EB/OL]. [2023-02-20]. https://arxiv.org/abs/2012.10069.
14	ZHU H Y, JIN Y C. Multi-objective evolutionary federated learning. IEEE Transactions on Neural Networks and Learning Systems, 2020, 31(4): 1310- 1322. doi: 10.1109/TNNLS.2019.2919699
15	QOLOMANY B, AHMAD K, AL-FUQAHA A, et al. Particle swarm optimized federated learning for industrial IoT and smart city services[C]//Proceedings of GLOBECOM'20. Washington D. C., USA: IEEE Press, 2020: 1-6.
16	CHEN F, LUO M, DONG Z H, et al. Federated meta-learning with fast convergence and efficient communication[EB/OL]. [2023-02-20]. https://arxiv.org/abs/1802.07876.pdf.
17	YAO X, SUN L F. Continual local training for better initialization of federated models[C]//Proceedings of 2020 IEEE International Conference on Image Processing. Washington D. C., USA: IEEE Press, 2020: 1736-1740.
18	WANG Z B, SONG M K, ZHANG Z F, et al. Beyond inferring class representatives: user-level privacy leakage from federated learning[C]//Proceedings of INFOCOM'19. Washington D. C., USA: IEEE Press, 2019: 2512-2520.
19	陈乃月, 金一, 李浥东, 等. 基于区块链的公平性联邦学习模型. 计算机工程, 2022, 48(6): 33- 41. URL
	CHEN N Y, JIN Y, LI Y D, et al. Federated learning model with fairness based on blockchain. Computer Engineering, 2022, 48(6): 33- 41. URL
20	温亚兰, 陈美娟. 融合联邦学习与区块链的医疗数据共享方案. 计算机工程, 2022, 48(5): 145-153, 161. URL
	WEN Y L, CHEN M J. Medical data sharing scheme combined with federal learning and blockchain. Computer Engineering, 2022, 48(5): 145-153, 161. URL
21	SHOHAM N, AVIDOR T, KEREN A, et al. Overcoming forgetting in federated learning on non-IID data[EB/OL]. [2023-02-20]. https://arxiv.org/abs/1910.07796.pdf.
22	LEE G, SHIN Y, JEONG M, et al. Preservation of the global knowledge by not-true self knowledge distillation in federatedlearning[EB/OL]. [2023-02-20]. https://arxiv.org/abs/2106.03097v2.
23	ALJUNDI R, BABILONI F, ELHOSEINY M, et al. Memory aware synapses: learning what (not) to forget[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 144-161.
24	ESTER M, KRIEGEL H P, SANDER J, et al. A density-based algorithm for discovering clusters in large spatial databases with noise[C]//Proceedings of National Conferences on Aritificial Intelligence. [S. l. ]: AAAI Press, 1996: 226-231.
25	KARIMIREDDY S, KALE S, MOHRI M, et al. Scaffold: stochastic controlled averaging for on-device federated learning[C]//Proceedings of the 37th International Conference on Machine Learning. [S. l. ]: PMLR, 2020: 1-10.
26	KOPPARAPU K, LIN E. FedFMC: sequential efficient federated learning on non-IID data[EB/OL]. [2023-02-20]. https://arxiv.org/abs/2006.10937.pdf.
27	CHEN Y, WANG L, XIN Q, et al. MetaFed: federated learning among federations with cyclic knowledge distillation for personalized healthcare[EB/OL]. [2023-02-20]. https://arxiv.org/abs/2206.08516.

[1]	马越, 温蜜. 基于多尺度LDTW和TCN的空间负荷预测方法[J]. 计算机工程, 2024, 50(3): 106-113.
[2]	张晓均, 李兴鹏, 唐伟, 郝云溥, 薛婧婷. 云-边融合的可验证隐私保护跨域联邦学习方案[J]. 计算机工程, 2024, 50(3): 148-155.
[3]	刘少杰, 文斌, 王泽旭. 基于联邦学习的多技术融合数据交易方法[J]. 计算机工程, 2024, 50(3): 182-190.
[4]	王丽娟, 邢津萍, 尹明, 郝志峰, 蔡瑞初, 温雯. 基于一致性图的权重自适应多视角谱聚类算法[J]. 计算机工程, 2024, 50(2): 122-131.
[5]	郑晨俊, 曾艳, 袁俊峰, 张纪林, 王鑫, 韩猛. 基于联邦学习的船舶AIS轨迹预测算法[J]. 计算机工程, 2024, 50(2): 298-307.
[6]	刘思慧, 高全学, 宋伟, 谢德燕. 基于加权张量低秩约束的多视图谱聚类[J]. 计算机工程, 2024, 50(1): 129-137.
[7]	张攀峰, 吴丹华, 董明刚. 基于粒子群优化的差分隐私深度学习模型[J]. 计算机工程, 2023, 49(9): 144-157.
[8]	江雨燕, 陶承凤, 李平. 数据增强和自适应自步学习的深度子空间聚类算法[J]. 计算机工程, 2023, 49(8): 96-103, 110.
[9]	郑美光, 杨泳. 基于互信息软聚类的个性化联邦学习算法[J]. 计算机工程, 2023, 49(8): 20-28.
[10]	李泽水, 冀俊忠, 杨翠翠. 基于边权重信息深度网络嵌入的PPIN功能模块检测[J]. 计算机工程, 2023, 49(8): 69-76.
[11]	邱天晨, 郑小盈, 祝永新, 封松林. 面向非独立同分布数据的联邦学习架构[J]. 计算机工程, 2023, 49(7): 110-117.
[12]	高小方, 原玉梁, 温静, 白雪飞. 面向相交多流形聚类的标签传播算法[J]. 计算机工程, 2023, 49(6): 90-98.
[13]	位雅, 张正军, 何凯琳, 唐莉. 基于相对密度的密度峰值聚类算法[J]. 计算机工程, 2023, 49(6): 53-61.
[14]	戴浩磊, 黄永慧, 周郭许. 基于超图正则化非负张量链分解的聚类分析[J]. 计算机工程, 2023, 49(6): 81-89.
[15]	温依霖, 赵乃良, 曾艳, 韩猛, 岳鲁鹏, 张纪林. 基于本地模型质量的客户端选择方法[J]. 计算机工程, 2023, 49(6): 131-143.

选择文件类型/文献管理软件名称

选择包含的内容