Federated Learning For Collaborative Optimization: Theory And Application

The performance of deep learning algorithms based on neural networks depends largely on sufficient and high-quality training data.However,data in the real world is usually generated in a distributed form and reside in different data owners,such as devices or institutions.Due to data sensitivity and various data policy constraints,data may not be directly gathered and integrated into the central server.Therefore,deep learning models face the problem of data silos.As a privacy-preserving distributed learning framework,federated learning is able to carry out joint modeling through parameter exchange under the encryption mechanism while keeping training data locally.Federated learning enables efficient collaborative training among multiple participants.It ensures information security and data privacy during communication,and thus becoming a reliable framework to balance data protection and application.In view of the challenges faced by federated learning and the shortcomings of existing related works,such as data heterogeneity,device heterogeneity,limited computing and communication resources,the research in this dissertation starts from the fundamental theory of federated learning and optimization,then focuses on modeling for various real scenarios and applications.This thesis gradually increases the difficulty of problems and attempts to solve challenging problems.The main works are summarized as follows:(1)Federated learning with two or more parties usually faces the challenge of differences in sample space,feature space,and label space among different institutions.To tackle this problem,this thesis proposes a multi-party dual learning method based on federated collaboration,to alleviate the problem of few overlapping samples and inconsistent feature space among multiple parties,and explicitly exploits the probabilistic correlation and structural symmetry between dual tasks to regularize the training process.The method introduces a feature-oriented differential privacy technique to avoid possible privacy leakage of raw features in the dual inference process,enabling each party to build flexible and powerful models separately,whose accuracy is no less than that of non-distributed self-learning approaches.The method achieves performance improvement compared with conventional multi-party learning methods that target to this problem,as demonstrated through simulations on realworld datasets.(2)Due to the wide usage of parameter averaging in model aggregation,the federated learning paradigm raises high requirements for aligned input features of models,which is hard to realize in complicated Io T scenarios with various monitoring indicators.Based on the model inversion,this thesis proposes a feature collaborative completion approach in federated scenarios to tackle this problem,and the method is tolerant of Io T devices with unaligned feature spaces.Local bilevel optimizations for both model parameters and input features are performed iteratively,in which the internal correlations of local sensor data provide additional guidance for the feature inference and completion,and thus building a unified feature space.Experimental results demonstrate that the proposed method improves the performance of the federated learning when facing missing features.(3)In traditional federated learning algorithms,the knowledge-sharing process based on parameter averaging among multiple participants is contradictory to human recognition and communication.Aiming at the limitation of deep learning in dealing with unseen classes of samples,this thesis proposes a federated zero-shot knowledge transfer method for cross-domain federated collaborative learning.Participants learn the mapping relationship between images and attributes locally,and share the structural description corresponding to the classes with other members.Participants can leverage the learned mapping function to obtain label-related attributes of samples from unseen classes,and make reasoning on the classes of these samples by combining attributes and the received structured description information.The method integrates deep learning and logical reasoning,and promotes the decision-making expansion of dependent knowledge in the federated environment.It improves the interpretability of the federated learning,and achieves high performance in extensive experiments.(4)Due to the quagmire of skewed data distribution across participants as well as the computation and communication bottleneck of local devices,determining how to build smaller customized models for clients in various scenarios while providing updates applicable to the global model remains a challenge.A data-driven federated heterogeneous differentiable sampling method is presented for a more effective and communication-efficient collaborative learning scheme with non-IID data.It distributes different subnets of the global model to clients for updating,and the adaptive sampling rates allow each client to extract optimal local architecture from the supernet according to its unique data properties which might be different from those for other clients.The presented method improves test accuracies on both global and local datasets while speeding up the convergence of the global model,and it reduces local computation and communication costs significantly,as demonstrated through experiments on real-world datasets.(5)Based on the above studies,this thesis analyzes and summarizes the properties of medical data and proposes a collaborative multimodal learning method for the diagnosis of COVID-19 based on federated learning.Medical images in a single site are usually of a limited amount or weakly labeled,while integrating data scattered around different institutions to build effective models is not allowed due to data policy restrictions.The collaborative multimodal diagnosis method seeks to effectively leverage heterogeneous data from multiple parties while preserving patients’ privacy.Specifically,a siamese branched network is introduced as the backbone of local models in medical institutions,to capture inherent relationships across heterogeneous samples.The model is capable of handling semi-supervised inputs in multimodalities and conducting task-specific training,in order to improve the model performance in various scenarios.The effectiveness of the method are demonstrated through extensive simulations on real-world COVID-19 datasets.