Article Details

Abstract

This paper addresses the lack of systematic evaluation of federated learning algorithms in practical applications. Using the lightweight platform LightFL, it compares the performance of 10 mainstream algorithms on the MNIST dataset under two data partitioning methods: IID and Dirichlet Non-IID. In the IID scenario, most algorithms perform well with minimal differences. However, in the highly heterogeneous Dirichlet Non-IID scenario, the robustness of each algorithm is challenged. SCAFFOLD, through its unique client-side bias correction mechanism, effectively mitigates the performance degradation caused by data heterogeneity, demonstrating good robustness. The research shows that the performance of federated learning algorithms is highly dependent on the characteristics of the data distribution. When the data distribution is relatively uniform, simple and efficient algorithms such as FedAvg can be given priority; however, in highly heterogeneous scenarios, algorithms with strong robustness, such as SCAFFOLD, are more reliable choices. These findings provide empirical evidence for the selection of federated learning algorithms in different scenarios and have reference value for their promotion and application.

Keywords

Federated Learning; algorithm evaluation; independent and identically distributed; non-independent and identically distributed

References

1. [1] MCMAHAN B，MOORE E，RAMAGE D，et al. Communication-Efficient Learning of Deep Networks from Decentralized Data [C]//Proceedings of the 20th International Conference on Artificial Intelligence and Statistics.Fort Lauderdale：PMLR，2017：1273-1282.
2. [2] Liu Ruoxuan, Gao Linghang, Xie Guoyun, etc Design and Implementation of Lightweight Federated Learning Platform LightFL [J]. Modern Information Technology, 2025, 9 (6): 71-74
3. [3] LI T，SAHU A K，ZAHEER M，et al. Federated Optimization in Heterogeneous Networks [C]//Proceedings of Machine Learning And Systems.Austin：MLSys，2020，2：429-450.
4. [4] HSU T M H，QI H，BROWN M. Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification [J/OL].arXiv:1909.06335 [cs.LG].[2025-09-06].https://doi.org/10.48550/arXiv.1909.06335.
5. [5] KARIMIREDDY S P，KALE S，MOHRI M，et al. SCAFFOLD: Stochastic Controlled Averaging for Federated Learning [C]//International Conference on Machine Learning.[S.I.]：PMLR，2020：5132-5143.
6. [6] WANG J，LIU Q，LIANG H，et al. Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization [C]//Advances in Neural Information Processing Systems 15.Vancouver：MIT Press，2020，33：7611-7623.
7. [7] REDDI S J，CHARLES Z，ZAHEER M，et al. Adaptive Federated Optimization [J/OL].arXiv:2003.00295 [cs.LG].[2025-09-03].https://doi.org/10.48550/arXiv.2003.00295.
8. [8] TAN Y，LONG G D，LIU L，et al. FedProto: Federated Prototype Learning across Heterogeneous Clients [C]//Proceedings of the AAAI Conference on Artificial Intelligence.Washington：AAAI Press，2022，36（8）：8432-8440.
9. [9] LI T，SANJABI M，BEIRAMI A，et al. Fair Resource Allocation in Federated Learning [C]//International Conference on Learning Representations.[S.I.]：OpenReview，2020.
10. [10] WANG H，YUROCHKIN M，SUN Y，et al. Federated Learning with Matched Averaging [J/OL].arXiv:1905.10497 [cs.LG].[2025-08-26].https://doi.org/10.48550/arXiv.1905.10497.
11. [11] LIN T，KONG L J，STICH S U，et al. Ensemble Distillation for Robust Model Fusion in Federated Learning [J/OL].arXiv:2006.07242 [cs.LG].[2025-08-29].https://doi.org/10.48550/arXiv.2006.07242.

Funding

2025 Beijing University of Information Science and Technology Innovation and Entrepreneurship Training Program for College Students (S202511232139); Support Program for Young Backbone Teachers of Beijing University of Information Science and Technology in 2024 (YBT202450); Development of an AI based weather forecasting model for the Qinghai Tibet Plateau region (S2426030); 2025 Beijing University of Information Science and Technology "Starlight Fund" Funding Project (XG2025PT92)

About the authors

Wang Jiaxin (2005-), female, Han ethnicity, from Sanmenxia, Henan Province, undergraduate studies, research direction: Federated Learning; Li Kehong (2004-), male, Tujia ethnic group, from Zhongxian County, Chongqing, undergraduate student, research direction: Federal Studies; Liu Suyue (2005-), female, Han ethnicity, from Shangqiu, Henan Province, undergraduate studies, research direction: Federated Learning; Wei Qijia (2006-), female, Han ethnicity, from Tongzhou, Beijing, undergraduate studies, research direction: Federated Learning; Yu Le (2005-), female, Han ethnicity, from Daxing, Beijing, undergraduate studies, research direction: Federated Learning; Corresponding author: Wang Can (1990-), male, Han ethnicity, from Jining, Shandong, associate professor, PhD, research direction: Federated Learning.