Federated Learning Under User-Level Differential Privacy

In recent years,the rapid development of mobile hardware has laid the foundation for AI application of large-scale computing on the one hand,and also led to more scattered data resources on the other hand.The analysis and query over the scattered data may leak the sensitive information of individuals,which leads to more and more people pay attention to their privacy.The leakage incidents of the privacy drive most of countries to enact powerful laws and regulations to protect the individuals’ privacy.In the technology field,compared with the traditional machine learning frameworks,federated learning can train some distributed optimal models as well as protect the individuals’ privacy.However,communication cost,security,and gradient leakage are still the serious issues in federated learning.As a common privacy protection method,differential privacy is often used to complete privacy calculation in machine learning.Among them,local differential privacy is naturally compatible with the idea of federated learning due to its decentralized implementation,but the introduction of noise reduces the usability of the model.In view of the above problems,the main contributions are as follows:First of all,since different data distribution leads to different contributions to the global model,we compute the sign similarity or cosine similarity between local updates and global updates.Based on the similarity,we can replace the current local updates with historical updates,which is useful to accelerate the model convergence.Secondly,we analyze the learning error caused by the Gauss mechanism,and figure out the changing trend of local updates.According to the error and the trend,we propose an efficient clipping strategy that is based on dynamic decay skill,to control the local sensitivity of updates.The basic composition theorem of differential privacy and Moments Accoutant mechanism are analyzed and extended.Finally,the convergence and privacy of the algorithm are analyzed and proved in the round paper theory.The experiment shows that the algorithm proposed in this paper has a small improvement in usability compared with the existing algorithm.In terms of communication cost,the number of training rounds required for the algorithm to achieve the target accuracy is reduced by 18% under the condition of non-independent and identically distributed data allocation;For data distribution with independent and identical distribution,the number of iteration rounds of the algorithm is about 1/3 of the existing algorithms.