A Research On Efficient Federated Learning Algorithms Based On Gradient Temporal Correlation

The boom in deep learning has been promoting data-driven approaches in all sorts of industries,but it has in turn spurred the concern of privacy protection of sensitive data.As a result,federated learning,as a promising alternative to traditional data-centralized learning has attracted much attention.Due to its nature of distributed optimization,however,the efficiency of federated learning remains to be a huge obstacle to its deployment and implementation,which has stimulated a wide range of research interests.To achieve efficient federated learning,this thesis focuses on the communication overhead and data heterogeneity in typical federated learning systems,and proposes corresponding solutions based on the observation and exploitation of gradient temporal structure.Specifically,for the uplink communication bottleneck in federated learning,a novel over-the-air computation based gradient transmission scheme,termed as TSA-GA,is proposed in this thesis,which reconstructs the compressed gradient aggregation using Bayesian compressed sensing method.To this end,the Turbo-CS algorithm,which was developed for sparse recovery,is adapted to serve for the reconstruction of sparse gradient signal related in time.The correlation(both in amplitude and support)is characterized by Markov chain models.To achieve reconstruction with low complexity,the computationally inefficient inter-round messages are approximated via Taylor expansion until the second order.Meanwhile,the hyper-parameters in these prior models are determined in real time by the Expectation Maximization(EM)algorithm.Thanks to the precise modeling of gradient prior structure,TSA-GA can greatly outperform the existing compression schemes that only rely on gradient sparsity in terms of the efficiency of gradient compression.Secondly,to alleviate overfitting during local update in federated learning with heterogeneous data across clients,a local update correction strategy,referred to as Fed Glob,is proposed.Specifically,the Fed Glob algorithm alleviates the inconsistency of local objective among the clients by introducing the global objective to the original local objective as a regularizer.Provided that the client only have access to its local data,it cannot calculate the gradient corresponding to the regular term.To address this problem,the global update direction of the previous round is adopted as the proxy that approximates the gradient direction corresponding to the current global objective(or regularizer),which is due to the temporal correlation of the gradient,i.e.,the global objective is decreasing even along the direction of the previous global update.Furthermore,the behavior of Fed Glob is analyzed through extensive experiments.As compared with the classical algorithm,Fed Glob can accelerate the convergence of federated learning and improve the generalization performance of the final model without changing the common communication protocol and model architecture of federated learning.