Research On Cross-Device Federated Learning For Heterogenous Edge Systems

With increasing stringent privacy and security regulations are being enacted and enforced,traditional cloud-centric machine learning training paradigm faces a new legal risk,and data barriers and data island have become a pressing challenge in the landing of artificial intelligence applications.Federated Learning(FL)is a novel privacy-preserving computing platform that provides a novel solution to overcome the data island barriers and boost the popularity of AI at the edge,but faces the challenge of edge heterogeneity instead.In the cross-device federation learning settings,the data collected by edge devices are usually affected by geographical distribution,cultural differences,device health,diurnal variations,learning task types,and other factors,increasing the statistical heterogeneity of training data.Therefore,it is meaningful to study adaptable deep learning algorithms in FL for edge heterogeneous scenarios.This thesis investigates the negative impact of each type of statistically heterogeneous data in edge heterogeneous scenarios on the federated training framework and the respective counter-strategies.This thesis classifies the types of statistical heterogeneity present in edge heterogeneous scenarios and designs distributed datasets with different types of statistical heterogeneity.To address the performance degradation and slow convergence issues of cross-device FL in statistically heterogeneous scenarios,we design a self-balancing FL framework and two efficient clustering federation learning frameworks.The proposed frameworks improve the robustness,efficiency and scalability of cross-device federated training from the perspective of global data imbalance,incongruent sub-optimization goals and distribution shift.Experimental results show that the proposed frameworks in the thesis can significantly improve the model accuracy,convergence speed and communication efficiency of FL in edge heterogeneous scenarios.The contents of this thesis are summarized as follows:(1)A self-balancing FL framework,Astraea,is designed to counter the negative effect of imbalanced federated training in edge heterogeneous scenarios.Specifically,this thesis explores for the first time the impact of various types of data imbalance on federated averaging and theoretically analyzes the phenomenon of global imbalance causing model accuracy degradation.This thesis proposes two strategies to mitigate the impact of global imbalance on FL: a global distribution-aware data augmentation strategy and a mediator-based training rescheduling strategy to mitigate the bias in the training procedure of global consensus models from both data and architecture aspects,respectively,and finally accelerate the model convergence and achieve better training efficiency and communication efficiency.(2)A FL framework Fed Group based on optimization similarity clustering is designed to prevent the impact of incongruent sub-optimization goals on federated training.This thesis proposes a novel FL framework based on Clustered Federal Learning(CFL)to improve its training efficiency in the presence of incongruent risk functions in the edges.To address the issue that the clustering measures used in previous CLF-based studies suffer from distance concentration phenomenon in high-dimensional space,this thesis designs two data-driven clustering measures MADC and EDC based on cosine similarity.Meanwhile,a newcomers’ cold start strategy is provided to enhance the scalability of Fed Group and the group can be initialized with only partial client participation.This thesis combines Fed Group with FL optimizer Fed Prox and discusses the connection between the model discrepancy and model accuracy.Experiments demonstrate that the proposed framework strikes a good balance between the computational overhead and the training efficiency.(3)A CFL-based framework Flex CFL for data distribution shifts is designed for addressing the problem of grouping rule decay due to the non-adaptability of CFL to local data distribution changes.Flex CFL provides a client-initiated migration strategy that can refresh the grouping rules without additional communication overhead,thus keeping the timeliness of the grouping rules.Meanwhile,Flex CFL leverages an inter-group aggregation strategy to explore the potential generalization improvement of the semi-pluralistic structure to the group consensus model.This thesis provides convergence proofs and ablation studies of Flex CFL,evaluates the communication efficiency,and verifies the effectiveness from theoretical and experimental aspects.Finally,based on several public datasets and benchmarks,this thesis provides evaluations of the proposed three FL frameworks.The experimental results show that Astraea can significantly improve the model accuracy on global imbalanced datasets and is communication efficient;Fed Group can achieve better clustering efficiency,reduce discrepancy and improve accuracy,with good scalability.Flex CFL shows significant superiority and communication efficiency in the presence of data distribution shift compared to other benchmarks.