Research On Efficient Federated Learning Algorithms For Data And System Heterogeneity

Federated Learning(FL)is an emerging distributed machine learning framework that allows edge devices to collaboratively train a shared global model without transferring their sensitive data to a central server.However,applying FL in practical scenarios faces major challenges of data heterogeneity and system heterogeneity.(1)Due to the differences in geographical location and usage of the devices participating in federated learning,the data collected between devices is non-independent and identically distributed(Non-IID).These heterogeneous data introduce unpredictable deviations to the global model,resulting in a decrease in model accuracy and a decrease in convergence speed;(2)Devices in the system have different physical characteristics such as computing power,energy consumption,and communication bandwidth.In the iterative process of federated learning that requires synchronous training,the heterogeneity of system increases the coordination overhead of server and terminal devices,prolongs the time of training rounds,and reduces training efficiency.At present,more and more people are paying attention to the heterogeneity of federated learning,but most of them are dedicated to solving one of data heterogeneity and system heterogeneity.In response to the above problems and challenges,this paper considers the impact of data heterogeneity and system heterogeneity on the federated learning system,explores an efficient federated learning algorithm for data and system heterogeneity,and proposes a Multi-Stage Semi-Asynchronous Federated Learning algorithm(MSSA-FL),the specific work is as follows:(1)Aiming at the problem of model accuracy degradation caused by heterogeneous data,this paper proposes a multi-stage training federated learning algorithm based on data completion.The data completion method solves the problem of data heterogeneity by supplementing heterogeneous training data between devices.However,current work of this kind often requires sacrificing data security and protection of data privacy,which violates the original intention of federated learning.In order to achieve the effect of data complementation while meeting the requirements of privacy protection,this paper proposes a multi-stage model training method based on the idea of "data not moving,model moving",that is,coordinating the model to train in multiple devices with complementary data distribution in order to achieve the purpose of complementing training data.In order to determine the complementary relationship between devices without additional data knowledge and realize the above model training process,this paper proposes a device combination algorithm.The algorithm clusters the equipment,uses the auxiliary dataset to infer the data distribution of the device cluster,and finally determines the complementary relationship between the device clusters to form a training group with data balance within the group.In order to maintain the effect of multi-stage training more effectively,this paper proposes a federated optimization algorithm based on the importance of parameters to alleviate the knowledge forgetting of the model during the training process.(2)Aiming at the problem of reduced training efficiency due to the heterogeneous characteristics of the system,this paper explores a new model update method and device selection strategy based on the characteristics of multi-stage training.Due to the heterogeneous nature of the system,the synchronous model update method seriously blocks the training process,which is the main reason for reducing the training efficiency of the system.To this end,this paper proposes a semi-asynchronous model update method,and uses an adaptive weighted aggregation algorithm to solve the problem of outdated models in the aggregation process.By allowing some models to be updated asynchronously,the model aggregation is more flexible,the aggregation frequency is increased,and non-blocking model training and updating are realized.Although the commonly used random device selection strategy is fair and simple,it does not make good use of the computing resources of powerful devices,and there is still a great potential to improve efficiency to be discovered.Therefore,this paper proposes a credit-limited fast device selection algorithm.This algorithm increases the possibility of fast equipment being selected by adjusting the selection probability of equipment.At the same time,the concept of credit points is introduced for devices,which prevents a small number of devices from being frequently selected,and ensures the fairness of device selection.Finally,this paper verifies the effectiveness of the proposed algorithm.The comparison results with FL algorithms in other works show that MSSA-FL has been improved in terms of model accuracy,model convergence speed and system work efficiency.Under specific experimental settings,our method achieves a 14.33%improvement in model accuracy and a 68.91% reduction in convergence time compared to the baseline method.