| Today,the amount of data generated and collected by 5G,the Internet of Things(Io T)and edge computing has exploded.In order to effectively utilize the massive data distributed in multiple data sources,federated learning proposed by Google researchers enables multiple participants to collaboratively train machine learning models.And the data is retained locally by the participating parties to avoid the privacy and security issues caused by traditional distributed machine learning aggregation of user data.At present,research on federated learning technology still faces the dual challenges of model personalization requirements and data heterogeneity.On the one hand,different model demanders of federated learning usually need to build different artificial intelligence businesses.These participants are not satisfied with the model structure designed uniformly by the center,but need to use their own independently designed model structure to participate in federated training.On the other hand,in the actual application of federated learning,in addition to the possible differences in the data distribution of the participants,the modalities of the participants’ data may also be different due to different data collection equipment and methods.The main research contents and contributions of this dissertation are as follows:(1)Model heterogeneity federated learning technology based on pseudo-label aggregation.For federated learning scenarios with heterogeneous model structures,when there are ample unlabeled natural samples available to represent the data characteristics of various participants,this dissertation proposes a federated learning method,Cotraining Federated Learning(CoFED).This method aligns the cognition of models from different participants on a public unlabeled natural dataset,enabling federated training that supports heterogeneous model structures.It is also compatible with heterogeneous tasks,models,and training processes.Compared to traditional methods,this algorithm decouples the models and training processes of different participants,allowing each participant to train their independently designed models using the best training method for their unique tasks.Additionally,CoFED protects the privacy of data,models,and training methods of all participants in a federated learning setting.This method achieves significant performance improvements for models with heterogeneous architectures in non-independent and identically distributed data scenarios.(2)Model heterogeneity federated learning technology based on Generative Adversarial Networks(GANs).The CoFED method’s applicability may be limited by the availability of public unlabeled datasets.Therefore,addressing scenarios lacking sufficient natural unlabeled samples,this dissertation introduces a complementary algorithm,Personalized Federated Learning via Generative Adversarial Network(PerFED-GAN),which utilizes locally trained models as discriminators to train Conditional Generative Adversarial Networks(CGANs).This process generates pseudo-samples containing local data features,enabling model cognition aggregation among participants through the exchange of pseudo-samples and sample aggregation with the center,thus supporting federated learning with heterogeneous model structures.Additionally,to prevent the leakage of local data privacy through generated samples,this dissertation introduces a differential privacy mechanism for the CGANs,further protecting participant data privacy.This method not only supports collaborative training among models with heterogeneous structures,achieving superior performance of personalized models,but also obviates the reliance on external datasets,effectively protecting participant data privacy and further extending the capabilities and practical scenarios of federated learning.(3)Modality heterogeneity federated learning technology based on Transformers.In response to the challenges of heterogeneous data modalities in federated learning,this dissertation proposes a federated learning training algorithm based on linear embedding and multimodal Transformers.This framework aims to design and comprehensively evaluate a new multimodal federated learning framework.Initially,it preprocesses different modal data into segment data conducive to subsequent processing through data segmentation and linear mapping.Then,it embeds corresponding data modality vectors to convey modality information.Subsequently,leveraging the strong capability of Transformers in representing different modalities and facilitating multimodal machine learning,it can learn and interpret data of various modalities,supporting federated learning training in scenarios with heterogeneous data modalities among different participants,while sharing knowledge among participants and improving model quality without compromising privacy.(4)Modality heterogeneity personalized federated learning technology based on Transformers.This dissertation delves into the inherent complexities of multimodal federated learning,with a particular focus on the differences in data modalities among participants and their personalized needs.The traditional federated learning paradigm,predicated on the assumption of data homogeneity,may encounter challenges in these settings.To surmount these challenges,this dissertation introduces a federated learning framework that effectively harnesses the advantages of Transformers in extracting and representing multimodal data,coupled with personalized federated learning strategies.This framework efficiently processes heterogeneous modal data,thereby achieving personalized multimodal federated learning.The multimodal personalized federated learning approach proposed in this dissertation contributes to the development of more reliable machine learning models,significantly expanding the applicability and practicality of the federated learning framework. |
