| Massive data has boosted the rapid development of artificial intelligence,and machine learning technology is broadly used in today’s production and daily life.Traditional machine learning usually stores the data centrally in the server for unified model training.However,the issue of data privacy has been widely concerned,and centralized training faces the difficulty of data collection and storage.As a technical means to solve the problem of “data island”,federated learning can fully protect the security of user data and enable users to collaborate efficiently on training at the same time.Federated learning is essentially a distributed machine learning framework,which keeps user data being local and enables users to benefit from each other in collaborative training.Federated learning faces the challenge of heterogeneous user data distribution in practical application scenarios.The participating user data is maintained by the data owner himself,and the data distribution difference between different users cannot be directly evaluated.In the case of heterogeneous data distribution,it shows problems such as performance degradation and unstable training process.Aiming at the above problems,this paper studies and improves the existing federated learning algorithms,and proposes two federated learning algorithms for heterogeneous data distribution:· Heterogeneous Federated Learning via Series of Historical Knowledge and Generative Learning(Fed SG): This algorithm uses the generative model to learn the model uploaded by the user and the local label counts,extracts the data distribu- tion in the global view as the global knowledge,and passes it to users through the generator.Users can enhance the local data by sampling from the generator,so as to weaken the difference of data distribution among users.The distillation of histor- ical knowledge is introduced into the local training of users to avoid forgetting the past learning results and reduce the disturbance of performance due to the decrease of user engagement.· Heterogeneous Federated Learning via Generative Learning and Mutual Dis-tillation(Fed GM): This algorithm extends the method of generative models to ex-tract global knowledge,and builds proxy dataset on the server side by the condi-tional generator,which are used by user models to perform mutual distillation onthe server side.The results obtained by mutual distillation are integrated into theglobal model instead of weighted average aggregation,which improves the effi-ciency of model convergence and at the same time protects user data privacy.In this thesis,we evaluate the above methods with extensive experiments on various datasets(i.e.,MNIST,EMNIST,CIFAR-10)and different heterogeneous settings.The proposed methods is guaranteed to converge under common assumptions,and achieves superior empirical accuracy in fewer communication runs than state-of-the-art methods. |
