Local Momentum Accelerated Based Non-IID Federated Learning Method

doi:10.19678/j.issn.1000-3428.0070254

Abstract

Abstract:

Federated Learning (FL), a distributed machine learning technology, has achieved significant results in privacy protection. However, in practical applications, client drift phenomena occur because of the Non-Independent and Identically Distributed (Non-IID) nature of data sources, leading to slow model convergence and performance degradation. To address this issue, this study proposes a Federated Local Momentum accelerated learning (FedLM) algorithm combined with the attention mechanism. FedLM introduces a global momentum term into local model updates, utilizing the global gradient information from previous rounds to smooth the current update process and correct the divergence of parameter update directions among heterogeneous clients, thereby reducing gradient oscillations and alleviating data heterogeneity issues. The attention mechanism dynamically adjusts the weight of each client in the global model update to improve the quality of the aggregation model. Experimental results show that FedLM achieves significantly better accuracy and stability than existing federated learning algorithms such as SCAFFOLD, FedCM, and Moon in image classification tasks with different levels of data heterogeneity, model structures, and datasets.

Key words: Federated Learning(FL), data heterogeneity, attention mechanism, client drift, local momentum

摘要：

联邦学习(FL)作为一种分布式机器学习技术, 在隐私保护方面取得了显著成果。然而, 在实际应用中, 由于数据源的非独立同分布(Non-IID)性, 导致客户端漂移现象, 从而引发模型收敛缓慢和性能下降问题。为此, 提出一种结合注意力机制的联邦本地动量加速学习(FedLM)算法。FedLM在本地模型更新中引入全局动量项, 利用前几轮的全局梯度信息来平滑当前的更新过程, 修正异构客户端的参数更新方向分歧, 从而减少梯度震荡, 缓解数据异构性问题。注意力机制则通过动态调整各客户端在全局模型更新中的权重, 以提升聚合模型的质量。实验结果表明, 在不同数据异构程度、不同模型结构以及不同数据集的图像分类任务中, FedLM的准确率和稳定性均显著优于现有的SCAFFOLD、FedCM、Moon等联邦学习算法。

关键词: 联邦学习, 数据异构, 注意力机制, 客户端漂移, 本地动量

YIN Hengjie, ZHENG Keqing, KE Jiannan, DONG Yunquan. Local Momentum Accelerated Based Non-IID Federated Learning Method[J]. Computer Engineering, 2026, 52(4): 103-110.

尹恒杰, 郑克清, 柯建楠, 董云泉. 基于本地动量加速的非独立同分布联邦学习方法[J]. 计算机工程, 2026, 52(4): 103-110.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0070254

https://www.ecice06.com/EN/Y2026/V52/I4/103

Figures/Tables 8

Fig.1 Client drift during Non-IID training

Fig.2 The data distribution of the top ten clients in the CIFAR-10 dataset under different heterogeneous settings

Fig.3 Accuracy of LeNet-5 on the CIFAR-10 dataset when α=0.1

Fig.4 Accuracy of LeNet-5 on the CIFAR-10 dataset when α=0.6

Fig.5 Accuracy of LeNet-5 on the CIFAR-100 dataset when α=0.1

Fig.6 Accuracy of LeNet-5 on the CIFAR-100 dataset when α=0.6

Fig.7 Accuracy of ResNet-18 on the CIFAR-100 dataset when α=0.1

Fig.8 Accuracy of ResNet-18 on the CIFAR-100 dataset when α=0.6

References 25

1	AASOUM N, JELLOULI I, AMJAD S, et al. Security and privacy-preserving techniques of federated learning in edge computing: a comparative study[C]//Proceedings of the 4th International Conference on Innovative Research in Applied Science, Engineering and Technology (IRASET). Washington D.C., USA: IEEE Press, 2024: 1-8.
2	ZHANG Y , ZHANG W L , PU L J , et al. To distill or not to distill: toward fast, accurate, and communication-efficient federated distillation learning. IEEE Internet of Things Journal, 2024, 11 (6): 10040- 10053. doi: 10.1109/JIOT.2023.3324666
3	吴若岚, 陈玉玲, 豆慧, 等. 抗攻击的联邦学习隐私保护算法. 计算机工程, 2025, 51 (2): 179- 187. doi: 10.19678/j.issn.1000-3428.0068705
	WU R L , CHEN Y L , DOU H , et al. Privacy preserving algorithm using federated learning against attacks. Computer Engineering, 2025, 51 (2): 179- 187. doi: 10.19678/j.issn.1000-3428.0068705
4	CHADDAD A , WU Y H , DESROSIERS C . Federated learning for healthcare applications. IEEE Internet of Things Journal, 2024, 11 (5): 7339- 7358. doi: 10.1109/JIOT.2023.3325822
5	DU J H , QIN N , HUANG D Q , et al. Lightweight FL: a low-cost federated learning framework for mechanical fault diagnosis with training optimization and model pruning. IEEE Transactions on Instrumentation and Measurement, 2024, 73, 3504014.
6	MIAO J X, YANG Z X, FAN L L, et al. FedSeg: class-heterogeneous federated learning for semantic segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2023: 8042-8052.
7	MA Z R , LIU Y , MIAO Y B , et al. FlGan: GAN-based unbiased federated learning under Non-ⅡD settings. IEEE Transactions on Knowledge and Data Engineering, 2024, 36 (4): 1566- 1581. doi: 10.1109/TKDE.2023.3309858
8	HIJAZI N M , ALOQAILY M , GUIZANI M , et al. Secure federated learning with fully homomorphic encryption for IoT communications. IEEE Internet of Things Journal, 2024, 11 (3): 4289- 4300. doi: 10.1109/JIOT.2023.3302065
9	LI T , SAHU A K , TALWALKAR A , et al. Federated learning: challenges, methods, and future directions. IEEE Signal Processing Magazine, 2020, 37 (3): 50- 60. doi: 10.1109/MSP.2020.2975749
10	KARIMIREDDY S P, KALE S, MOHRI M, et al. SCAFFOLD: stochastic controlled averaging for federated learning[C]//Proceedings of the 37th International Conference on Machine Learning. Washington D.C., USA: IEEE Press, 2020: 5132-5143.
11	宋华伟, 李升起, 万方杰, 等. 非独立同分布场景下的联邦学习优化方法. 计算机工程, 2024, 50 (3): 166- 172. doi: 10.19678/j.issn.1000-3428.0067791
	SONG H W , LI S Q , WAN F J , et al. Federated learning optimization method in Non-ⅡD scenarios. Computer Engineering, 2024, 50 (3): 166- 172. doi: 10.19678/j.issn.1000-3428.0067791
12	LIU W , CHEN L , CHEN Y F , et al. Accelerating federated learning via momentum gradient descent. IEEE Transactions on Parallel and Distributed Systems, 2020, 31 (8): 1754- 1766. doi: 10.1109/TPDS.2020.2975189
13	BRAUWERS G , FRASINCAR F . A general survey on attention mechanisms in deep learning. IEEE Transactions on Knowledge and Data Engineering, 2023, 35 (4): 3279- 3298. doi: 10.1109/TKDE.2021.3126456
14	LIU Y L, GAO Y, YIN W T. An improved analysis of stochastic gradient descent with momentum[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2020: 18261-18271.
15	ZHOU H H, LAN T, VENKATARAMANI G, et al. Every parameter matters: ensuring the convergence of federated learning with dynamic heterogeneous models reduction[EB/OL]. [2024-06-05]. https://arxiv.org/abs/2310.08670.
16	SABUG L , FAGIANO L , RUIZ F . Sample-based trust region dynamics in contextual global optimization. IEEE Control Systems Letters, 2024, 8, 1619- 1624. doi: 10.1109/LCSYS.2024.3414970
17	MCMAHAN H B, MOORE E, RAMAGE D, et al. Communication-efficient learning of deep networks from decentralized data[EB/OL]. [2024-06-05]. https://arxiv.org/abs/1602.05629.
18	YAN B J, LIU B Y, WANG L J, et al. FedCM: a real-time contribution measurement method for participants in federated learning[C]//Proceedings of the International Joint Conference on Neural Networks (IJCNN). Washington D.C., USA: IEEE Press, 2021: 1-8.
19	LI Q B, HE B S, SONG D. Model-contrastive federated learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2021: 10708-10717.
20	GAO L, FU H Z, LI L, et al. FedDC: federated learning with Non-ⅡD data via local drift decoupling and correction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2022: 10102-10111.
21	KIM G, KIM J, HAN B. Communication-efficient federated learning with accelerated client gradient[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2024: 12385-12394.
22	SUN Y, SHEN L, HUANG T S, et al. FedSpeed: larger local interval, less communication round, and higher generalization accuracy[EB/OL]. [2024-06-05]. https://arxiv.org/abs/2302.10429.
23	LECUN Y , BOTTOU L , BENGIO Y , et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998, 86 (11): 2278- 2324. doi: 10.1109/5.726791
24	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2016: 770-778.
25	IQBAL M , LI R J , XU J Y . Behavior analysis of photo-taking tourists at attraction-level using deep learning and latent Dirichlet allocation in conjunction with kernel density estimation. IEEE Access, 2024, 12, 92945- 92959. doi: 10.1109/ACCESS.2024.3395469

[1]	LI Jiao, FAN Haodong, HONG Xudong, XU Zhenyi, FAN Xu, HUANG Jun. Multi-Label Image Classification Based on Label Visual Prototype Learning [J]. Computer Engineering, 2026, 52(4): 229-238.
[2]	TANG Weibo, FANG Qiang, LI Peigen, AI Longjin, XIONG Jinhong, XIA Haiting. RSD-YOLO-Based Small Target Detection in UAV Aerial Images [J]. Computer Engineering, 2026, 52(4): 214-228.
[3]	CAO Jiwei, LUO Fei, DING Weichao. BS-YOLO: A Small Object Detection Algorithm Based on BSAM Attention Mechanism and SCConv [J]. Computer Engineering, 2026, 52(3): 119-127.
[4]	CHEN Guolian, FENG Ziyang, CAO Junkuo. Research on Cyberbullying Detection Based on Multimodal Spatial Feature Fusion [J]. Computer Engineering, 2026, 52(3): 255-263.
[5]	PAN Lihu, YIN Jiali, ZHANG Rui, XIE Binhong, ZHANG Linliang. Global-Local Spatiotemporal Perception Model for Traffic Flow Prediction [J]. Computer Engineering, 2026, 52(3): 392-402.
[6]	SU Jianhua, CHI Yunxian, XU Yunfeng, GAO Kai. Multimodal Intent Recognition Based on Attention Modality Fusion [J]. Computer Engineering, 2026, 52(3): 234-242.
[7]	WU Xuesong, CHEN Yuanyuan, ZHOU Tao. Adaptive No-Reference Image Quality Assessment Based on Multi-Scale Pyramid Pooling [J]. Computer Engineering, 2026, 52(3): 107-118.
[8]	LIU Xiaoyu, LIAO Zhifang, TAN Sui, YU Zhiwu. Bridge Dynamic Strain Prediction Based on Stacked GRU Neural Network [J]. Computer Engineering, 2026, 52(3): 441-450.
[9]	WEN Lang, GOU Guanglei, BAI Ruifeng, MIAO Wanyu. Few-shot Fine-grained Image Classification Based on Neighborhood Fusion and Feature Enhancement [J]. Computer Engineering, 2026, 52(2): 158-166.
[10]	SONG Chaoqi, LIU Ying, HE Jinglu, LI Daxiang. Few-shot Image Classification Method Based on Salient Position Interaction Transformer [J]. Computer Engineering, 2026, 52(2): 167-176.
[11]	WANG Qingrong, HAO Fule, ZHU Changfeng, WANG Junjie. Research on Vehicle Trajectory Prediction Based on Multifeature Fusion [J]. Computer Engineering, 2026, 52(2): 331-341.
[12]	XU Xiaoyang, WEI Wei, GAO Chongyang. Infrared Ship Target Detection Based on Improved YOLOv7-tiny [J]. Computer Engineering, 2026, 52(2): 209-220.
[13]	XUE Yang, QIN Yao, ZHANG Shuxiang. Collaborative Recommendation Based on Graph Neural Network Clustering Subgraphs Using Dual Graph Attention Network [J]. Computer Engineering, 2026, 52(2): 89-100.
[14]	LIU Chang, LIANG Bingxue, TIAN Rongkun, QIN Yuhua. Medical and Health Question Classification Based on Multi-feature Fusion and Hybrid Neural Network [J]. Computer Engineering, 2026, 52(2): 342-355.
[15]	DAN Chonghong, WEI Honglei, HE Zhou, WU Guanfeng. SRMpose: Multi-Scale Feature Extraction Keypoint Detection Algorithm [J]. Computer Engineering, 2026, 52(2): 136-147.

Please choose a citation manager

Content to export