Research On Fedrated Learning For Heterogeneous Environment

With the rapid development of Io T and mobile computing technologies,there are more and more machine learning systems changing from traditional centralized training architectures to distributed training architectures;on the other hand,laws and regulations to protect private data,such as the General Data Protection Regulation,are proposed worldwide,and they require machine learning systems not only to have strong performance,but also must pay attention to private data security issues.In this context,federated learning as a distributed machine learning architecture that protects the privacy of users’ private data has become the focus of attention in academia and industry.Federated learning systems aggregate users’ local models into global models to obtain better model performance,and users only need to upload model parameters without sharing private data.In real-life scenarios,the running platforms of users’ model,the data distribution of users’ private data,and the channel environment for users’ communication in federated learning systems are often heterogeneous.However,traditional federated learning frameworks are designed under the assumption that users’ platforms,private data,and communication channels are homogeneous,and thus do not work properly in the face of heterogeneous environments.Heterogeneous user platforms lead to heterogeneous user local models,and the global model cannot be calculated by weighted aggregation because the model parameters are not aligned;heterogeneous user private data lead to a single global model that cannot match each user’s private data well,resulting in a reduction of the average model accuracy;heterogeneous communication channels lead to a reduction of the communication overhead upper limit,and when the system’s communication overhead is larger than the channel’s ability,the training process of federation learning is forced to pause.In order to solve the above problems,this paper conducts a series of researches on the federal learning framework for heterogeneous environments.Three improved federation learning frameworks based on knowledge distillation,multi-task learning and hypernetworking techniques are respectively proposed to address three challenges: platform heterogeneity,private data heterogeneity and communication channel heterogeneity,respectively.The main contributions of this paper are summarized as follows.1.To address the problem of heterogeneous user platforms,a framework based on knowledge distillation is proposed to achieve model knowledge transfer.Knowledge distillation abstracts the knowledge on the heterogeneous model as models’ soft predictions.Soft prediction is model’s probabilistic judgment on which label the input data belongs to.The set of soft predictions contains the knowledge of the model and is independent of the model structure.In this paper,the aggregation of heterogeneous models is accomplished indirectly through the weighted aggregation and training of soft predictions.Facing the reduction of model accuracy due to unhelpful knowledge migration among models,this paper proposes the knowledge coefficient matrix,which can help users distinguish the contribution of other users’ model soft predictions to their local model performance.Knowledge coefficient matrix can also ensure more efficient collaboration among users with similar private data distribution in model training.2.To address the problem of heterogeneous user private data,a framework based on task decomposition techniques in multi-task learning is proposed.In this paper,the model parameters of users are decomposed into global model parameters shared by all users and task-specific model parameters unique to each user.The shared global model parameters learn and utilize the common knowledge of all users,while the taskspecific model parameters are obtained by dividing the original task into several subtasks using task decomposition techniques.The task decomposition is based on the distribution of users’ private data,and the aggregation of subtask model parameters is performed only among users with the same label of data,which prevents harmful global model aggregation among users with unrelated data due to private data heterogeneity and improves the accuracy of the model on users’ local test datasets.3.To address the problem of heterogeneous user communication channels,a framework based on hypernetworks is proposed to reduce users’ communication overhead.Hypernetworks can quantify the importance of local model parameters in layer-wised.After ranking layers’ importance,we can only transmit those layers that contribute more to the model performance,saving communication overhead when the user’s communication channel environment becomes poor;and when the user’s communication channel environment is good,our framework can also learn the optimal model aggregation weights at the layer level to achieve personalized layerwised model aggregation for users’ model.Finally achieving the goal of dynamically adapting to the user’s heterogeneous communication channel while maintaining high accuracy model performance.