Personalized Federated Learning Algorithm Based on Mutual Information and Soft Clustering

doi:10.19678/j.issn.1000-3428.0066689

Abstract

Abstract:

Federated learning is a distributed machine learning technique for collaboratively training machine learning models for multiple clients while protecting the privacy of client data. However, the heterogeneity inherent in client data limits the full application potential of federated learning, for which personalized federated learning is a viable solution. The traditional clustering-based personalized federated learning schemes group clients with the same data distribution into one cluster, exploiting the homogeneous nature of some client data and reducing the impact of data heterogeneity on federated learning; however, this approach fails to account for the possibility of clients belonging to multiple clusters. Based on the concept that client data approximate adhere to multiple data distributions, a personalized Federated learning algorithm is proposed based on Mutual information and Soft clustering(pFedMS).A mutual information formula based on model features is introduced to address the shortcomings of current federated learning client clustering indices, which can not accurately reflect the similarity of model features.This formula serves as a clustering index that effectively distinguishes similar clients. A clustering rationality measurement method based on intra-class and inter-class distances is proposed to dynamically adjust the clustering results. The similarity between clients and clusters is calculated using affiliation, which allows clients to belong to multiple clusters simultaneously and improves the performance of the clustering algorithm. Experimental results on CIFAR-10 and Fashion-MNIST(FMNIST) datasets show that the pFedMS improves the Best Mean Testing Accuracy(BMTA) of clients by 2.4 to 3.0 percentage points compared to the comparison algorithms such as FedAvg, CFL.

Key words: personalized federated learning, data bias, soft clustering, model feature, mutual information

摘要：

联邦学习是一种为多个客户协作训练机器学习模型的分布式机器学习技术，同时能够保护客户数据隐私，但客户数据异构性限制了联邦学习的应用，对此，个性化联邦学习是一种可行的解决方案。传统基于聚类的个性化联邦学习方案将具有相同数据分布的客户划分为一个集群，利用部分客户数据同构的特点减少了数据异构对联邦学习的影响，但忽略了客户属于多个集群的可能性。基于客户数据近似服从多种数据分布的思想，提出基于互信息软聚类的个性化联邦学习算法(pFedMS)。针对目前联邦学习客户聚类指标无法准确反映模型特征相似性的不足，给出基于模型特征的互信息公式作为聚类指标，有效区分相似客户；提出基于类内距离和类间距离的聚类合理性衡量方法，用于动态调整聚类结果；根据隶属度计算客户与集群的相似性，允许客户同时属于多个集群，提高聚类算法的性能。在CIFAR-10和FMNIST数据集上的实验结果表明，pFedMS算法相较于FedAvg、CFL等对比算法，客户最高平均测试准确率提高了2.4~3.0个百分点。

关键词: 个性化联邦学习, 数据偏差, 软聚类, 模型特征, 互信息

Meiguang ZHENG, Yong YANG. Personalized Federated Learning Algorithm Based on Mutual Information and Soft Clustering[J]. Computer Engineering, 2023, 49(8): 20-28.

郑美光, 杨泳. 基于互信息软聚类的个性化联邦学习算法[J]. 计算机工程, 2023, 49(8): 20-28.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0066689

https://www.ecice06.com/EN/Y2023/V49/I8/20

Figures/Tables 10

References 36

1	POUYANFAR S, SADIQ S, YAN Y L, et al. A survey on deep learning. ACM Computing Surveys, 2019, 51 (5): 1- 36.
2	KONEČNÝ J, MCMAHAN H B, RAMAGE D, et al. Federated optimization: distributed machine learning for on-device intelligence[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1610.02527.
3	YANG Q, LIU Y, CHEN T, et al. Federated machine learning: concept and applications. ACM Transactions on Intelligent Systems and Technology, 2019, 10 (2): 1- 19.
4	DIAO E M, DING J, TAROKH V. HeteroFL: computation and communication efficient federated learning for heterogeneous clients[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2010.01264.
5	杨强, 刘洋, 程勇. 联邦学习. 北京: 电子工业出版社, 2020.
	YANG Q, LIU Y, CHENG Y. Federated learning. Beijing: Publishing House of Electronics Industry, 2020.
6	MCMAHAN B, MOORE E, RAMAGE D, et al. Communication-efficient learning of deep networks from decentralized data[C]//Proceedings of the 20th International Conference on Artificial Intelligence and Statistics. Ft. Lauderdale, USA: PMLR Press, 2017: 1273-1282.
7	ZHAO Y, LI M, LAI L Z, et al. Federated learning with non-IID data[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1806.00582.
8	DIAO E, DING J, TAROKH V. SemiFL: communication efficient semi-supervised federated learning with unlabeled clients[EB/OL]. [2023-01-02]. https://arxiv.org/abs/12106.01432.
9	MELIS L, SONG C Z, DE CRISTOFARO E, et al. Exploiting unintended feature leakage in collaborative learning[C]//Proceedings of IEEE Symposium on Security and Privacy. Washington D. C., USA: IEEE Press, 2019: 691-706.
10	KULKARNI V, KULKARNI M, PANT A. Survey of personalization techniques for federated learning[C]//Proceedings of the 4th World Conference on Smart Trends in Systems, Security and Sustainability. Washington D. C., USA: IEEE Press, 2020: 794-797.
11	DINH C T, TRAN N H, NGUYEN T D. Personalized federated learning with Moreau envelopes[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2020: 21394-21405.
12	FALLAH A, MOKHTARI A, OZDAGLAR A. Personalized federated learning with theoretical guarantees: a model-agnostic meta-learning approach. Advances in Neural Information Processing Systems, 2020, 33, 3557- 3568.
13	KHODAK M, BALCAN M F F, TALWALKAR A S. Adaptive gradient-based meta-learning methods. Advances in Neural Information Processing Systems, 2019, 32, 5917- 5928.
14	HANZELY F, RICHTÁRIK P. Federated learning of a mixture of global and local models[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2002.05516.
15	DENG Y Y, KAMANI M M, MAHDAVI M. Adaptive personalized federated learning[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2003.13461.
16	SHAMSIAN A, NAVON A, FETAYA E, et al. Personalized federated learning using hypernetworks[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2103.04628.
17	LI T, SAHU A K, ZAHEER M, et al. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2020, 2, 429- 450.
18	SHOHAM N, AVIDOR T, KEREN A, et al. Overcoming forgetting in federated learning on non-IID data[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1910.07796.
19	YAO X, SUN L F. Continual local training for better initialization of federated models[C]//Proceedings of 2020 IEEE International Conference on Image Processing. Washington D. C., USA: IEEE Press, 2020: 1736-1740.
20	LI D L, WANG J P. FedMD: heterogenous federated learning via model distillation[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1910.03581.
21	HUI Z Z, CHEN D J, XU Z H. Federation learning optimization using distillation[C]//Proceedings of 2021 Asia-Pacific Conference on Communications Technology and Computer Science. Washington D. C., USA: IEEE Press, 2021: 25-28.
22	ZHU Z D, HONG J Y, ZHOU J Y. Data-free knowledge distillation for heterogeneous federated learning[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2105.10056.
23	SATTLER F, MÜLLER K R, SAMEK W. Clustered federated learning: model-agnostic distributed multitask optimization under privacy constraints. IEEE Transactions on Neural Networks and Learning Systems, 2020, 32 (8): 3710- 3722.
24	BRIGGS C, FAN Z, ANDRAS P. Federated learning with hierarchical clustering of local updates to improve training on non-IID data[C]//Proceedings of International Joint Conference on Neural Networks. Washington D. C., USA: IEEE Press, 2020: 1-9.
25	WANG H, KAPLAN Z, NIU D, et al. Optimizing federated learning on non-IID data with reinforcement learning[C]//Proceedings of 2020 IEEE Conference on Computer Communications. Washington D. C., USA: IEEE Press, 2020: 1698-1707.
26	ZHANG M, SAPRA K, FIDLER S, et al. Personalized federated learning with first order model optimization[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2012.08565v4.
27	LUO J, WU S. Adapt to adaptation: learning personalization for cross-silo federated learning[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2110.08394v1.
28	UDDIN M P, XIANG Y, LU X Q, et al. Mutual information driven federated learning. IEEE Transactions on Parallel and Distributed Systems, 2021, 32 (7): 1526- 1538.
29	CHEN N Y, LI Y L, LIU X J, et al. A mutual information based federated learning framework for edge computing networks. Computer Communications, 2021, 176, 23- 30. doi: 10.1016/j.comcom.2021.05.013
30	FINN C, ABBEEL P, LEVINE S. Model-agnostic meta-learning for fast adaptation of deep networks[C]//Proceedings of International Conference on Machine Learning. New York, USA: PMLR Press, 2017: 1126-1135.
31	HINTON G, VINYALS O, DEAN J. Distilling the knowledge in a neural network[EB/OL]. [2023-01-02]. https://arxiv.org/abs/1503.02531.
32	DUAN M M, LIU D, JI X Y, et al. FedGroup: efficient clustered federated learning via decomposed data-driven measure[EB/OL]. [2023-01-02]. https://arxiv.org/abs/2010.06870.
33	CANG S, YU H N. Mutual information based input feature selection for classification problems. Decision Support Systems, 2012, 54 (1): 691- 698. doi: 10.1016/j.dss.2012.08.014
34	王树芬, 张哲, 马士尧, 等. 一种鲁棒的半监督联邦学习系统. 计算机工程, 2022, 48 (6): 107-114, 123 URL
	WANG S F, ZHANG Z, MA S Y, et al. A robust semi-supervised federated learning system. Computer Engineering, 2022, 48 (6): 107-114, 123 URL
35	LI Q B, HE B S, SONG D. Model-contrastive federated learning[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 10708-10717.
36	KORNBLITH S, NOROUZI M, LEE H, et al. Similarity of neural network representations revisited[C]//Proceedings of International Conference on Machine Learning. New York, USA: PMLR Press, 2019: 3519-3529.

分类	文献来源	方法	优点	缺点
单一客户	文献[12-13]	利用元学习方法学习性能良好的初始全局模型，并适应数据异构的客户	能获取客户通用的全局模型	客户每次训练都需要自适应全局模型
	文献[14-15]	混合全局模型和本地模型，并通过控制混合程度实现混合模型个性化	利用了本地模型学到的个性化知识和全局模型学到的全局知识	当异构程度较大时，无法从全局模型中学到有用的全局知识
	文献[16]	通过个性化联邦超网络实现参数共享，并利用个性化参数实现客户模型个性化	客户通过共享通用参数即可实现本地模型个性化	共享参数和超网络存在隐私问题
	文献[17-19]	引入损失函数正则化项优化本地模型，正则项用于学习个性化知识	在损失函数层面限制本地更新，适用于异构场景	正则项参数需要多次优化
	文献[20-22]	利用知识蒸馏技术学习全局模型的知识，实现客户模型个性化	缓解数据分散引起的异构问题，个性化模型能实现更好的鲁棒性	知识蒸馏提高了客户的计算成本
同构客户集群	文献[23-25]	依据客户训练梯度，通过最优双分区、层次聚类等方法对客户进行聚类，并选择客户子集进行训练	利用客户部分同构的特点进行客户聚类，减轻了全局模型失效的问题	聚类指标有待优化，未考虑客户属于多个类的可能性
同构客户集群	文献[26-27]	客户下载其他客户的模型，计算最优模型组合进行本地训练，获得个性化模型	与部分客户进行协作，减轻了全局模型失效的问题	增加了客户的通信成本，存在隐私泄露的风险
	文献[28-29]	基于互信息方法优化联邦学习过程，实现有效全局模型聚合，并获得客户的个性化模型	引入互信息计算模型相似性，有效解决了个性化问题	给出的互信息无法准确反映模型特征

分类	文献来源	方法	优点	缺点
单一客户	文献[12-13]	利用元学习方法学习性能良好的初始全局模型，并适应数据异构的客户	能获取客户通用的全局模型	客户每次训练都需要自适应全局模型
	文献[14-15]	混合全局模型和本地模型，并通过控制混合程度实现混合模型个性化	利用了本地模型学到的个性化知识和全局模型学到的全局知识	当异构程度较大时，无法从全局模型中学到有用的全局知识
	文献[16]	通过个性化联邦超网络实现参数共享，并利用个性化参数实现客户模型个性化	客户通过共享通用参数即可实现本地模型个性化	共享参数和超网络存在隐私问题
	文献[17-19]	引入损失函数正则化项优化本地模型，正则项用于学习个性化知识	在损失函数层面限制本地更新，适用于异构场景	正则项参数需要多次优化
	文献[20-22]	利用知识蒸馏技术学习全局模型的知识，实现客户模型个性化	缓解数据分散引起的异构问题，个性化模型能实现更好的鲁棒性	知识蒸馏提高了客户的计算成本
同构客户集群	文献[23-25]	依据客户训练梯度，通过最优双分区、层次聚类等方法对客户进行聚类，并选择客户子集进行训练	利用客户部分同构的特点进行客户聚类，减轻了全局模型失效的问题	聚类指标有待优化，未考虑客户属于多个类的可能性
同构客户集群	文献[26-27]	客户下载其他客户的模型，计算最优模型组合进行本地训练，获得个性化模型	与部分客户进行协作，减轻了全局模型失效的问题	增加了客户的通信成本，存在隐私泄露的风险
	文献[28-29]	基于互信息方法优化联邦学习过程，实现有效全局模型聚合，并获得客户的个性化模型	引入互信息计算模型相似性，有效解决了个性化问题	给出的互信息无法准确反映模型特征

数据集	训练集图像数量/张	测试集图像数量/张	标签数量/个	图像分辨率/像素
FMNIST	60 000	10 000	10	28×28
CIFAR-10	50 000	10 000	10	32×32

数据集	训练集图像数量/张	测试集图像数量/张	标签数量/个	图像分辨率/像素
FMNIST	60 000	10 000	10	28×28
CIFAR-10	50 000	10 000	10	32×32

数据集	最高平均测试准确率
数据集	C=2	C=3	C=4	C=5
FMNIST	96.97	97.28	96.88	96.50
CIFAR-10	74.78	75.29	74.65	74.18

Please choose a citation manager

Content to export