Research On Efficient Federal Learning Methods On Non-IID Data

With the exponential rise of data from smart devices and sensors,privacy issue has become a growing concern.Federated learning(FL),a distributed machine learning framework,has achieved a breakthrough in preserving data privacy.In the vanilla FL approach,devices share weights or gradients of the locally trained models,rather than direct exchanging private raw data among devices.Those model parameters are uploaded to a server for aggregation and then sent back to the devices.While FL is an attractive privacy-preserving approach,challenges arise when FL is applied over wireless networks.First,limited bandwidth resources become the bottleneck for implementing FL.As model weights and gradients involve millions of bits,their repetitive transmission consumes considerable resources and causes considerable delays in the wireless network.Second,data distributions among devices are non-independent and identically distributed(non-IID),due to different local environments and characteristics of devices.Non-IID data induces weight divergence of the local models,resulting in degraded performance and sluggish convergence of the global model.This dissertation is dedicated to addressing the FL challenges of limited bandwidth resources and non-IID data,and the contribution of this dissertation is summarized as follows:(1)A Federated data augmentation algorithm,Conditional Variational Auto Encoderbased Federated Distillation,CvaeFDCvaeFD is proposed to achieve FL on Non-IID data with privacy preservation and communication efficiency.In CvaeFD,the Knowledge Distillation Mechanism is introduced to achieve Federated learning,through which knowledge is shared,rather than model parameters or gradients.The knowledge is designed based on hidden-layer features to reduce communication overhead and protect raw data privacy.In CvaeFD,the Conditional Variational Auto Encoder(CVAE)is adopted to generate the missing samples on Non-IID datasets with data characteristics of the Concept Shift,Feature Distribution Skew,Label Distribution Skew,and Quality Skew.Meanwhile,to generate cross-class samples that are easy to classify,the latent variables in CVAE are constrained and the attention mechanism is introduced.(2)A Federated data classification algorithm,Inter-Class Correlation Federated Distillation,IceFDIceFD is proposed to achieve FL on Non-IID data with privacy preservation and communication efficiency.In IceFD,the knowledge with much fewer parameters than model weights is transferred for communication efficiency.IceFD is utilized to address the non-IID data challenge with data characteristics of the Label Distribution Skew and Quality Skew.The knowledge is designed based on the Self-Attention Mechanism to extract the Inter-Class Correlation map,which reveals the correlation between every two classes.In IceFD,the analysis of the gradients demonstrates that students comprehensively learn the teacher’s knowledge in conjunction with their own understanding,rather than mimicking global knowledge entirely.(3)A Federated data classification algorithm with the Class-Specific and DeviceInvariant knowledge Federated Distillation,SparkFDSparkFD is proposed to achieve FL on Non-IID data with privacy preservation and communication efficiency.In SparkFD,The logits-based knowledge with much fewer parameters than model weights is transferred for communication efficiency.SparkFD is utilized to address the non-IID challenge with data characteristics of the Concept Shift and Feature Distribution Skew.The knowledge is extracted based on supervised contrastive learning and adversarial learning to gain the class-specific and device-invariant properties,i.e.,the most distinctive features of each class without being affected by the data distribution on each device.To extract the knowledge with the class-specific and device-invariant properties,a supervised contrastive federated learning process and a discriminative learning process are designed.(4)A Two-step Federated learning framework for data classification,Fed2KDFed2KD is proposed to achieve FL on Non-IID data with privacy preservation and communication efficiency.The knowledge is designed based on knowledge distillation.The outputs of the neural networks(logits)are utilized as knowledge.The knowledge occupied fewer parameters compared with the weights or gradients of neural networks.And the sharing of knowledge between devices and the server can achieve communication efficiency.This framework is designed for the Non-IID data scenarios with characteristics of the Concept Shift,Feature Distribution Skew,Label Distribution Skew,and Quality Skew.The first step is federated data augmentation,and the second step is federated data classification.Fed2 KD boosts classification accuracy through privacy-preserving data generation while improving communication efficiency through a new knowledge distillation scheme.To further improve the accuracy in non-IID data scenarios,a detection and discard mechanism is designed.The generalization ability of Fed2 KD is analyzed from the view of domain adaption to verify the effectiveness of this framework.