Differential Privacy-Based Horizontal Device-Cloud Federated Learning

With the advent of the 5G network era,machine learning has become an essential aspect of daily lives.Federated learning has emerged as a novel distributed training paradigm for largescale deep neural networks,which preserves the privacy of the participants’ data by keeping it on their local devices and sharing only the model updates with the central server.In doing so,it addresses the inherent privacy concerns arising from data sharing,which could otherwise lead to privacy violations.However,the distributed nature of federated learning raises new privacy concerns and challenges.Researches have revealed that the privacy of data and models remains at risk since existing federated learning protocols still have privacy vulnerabilities that can be exploited by potential malicious attackers both within and outside the system.Furthermore,the precision and efficiency of federated learning are also impacted because it necessitates frequent communication and massive data transmission,which can cause data transmission failure or delay due to the constrained bandwidth and unreliable network connections of mobile devices such as smartphones and tablets.Even though privacy protection can be achieved by directly applying differential privacy to federated learning,the high dimensionality of deep learning model weights and the frequent query iterations can cause the privacy budget to explode and thus affect the accuracy of the model.To address the aforementioned challenges,we propose two design schemes for different federated learning scenarios and demonstrates the effectiveness of the proposed framework on MNIST and CIFAR-10 datasets respectively.The main work and contributions of this thesis include:(1)To address the current privacy and communication challenges of horizontal devicecloud federated learning,we propose an algorithm based on gradient compression and differential privacy.In the algorithm,we incorporate the Top-k method with Federated Averaging algorithm to allocate the privacy budget to the k dimensions with the largest absolute value,thus boosting the privacy budget allocated to each dimension and improving accuracy.In order to select an appropriate noise addition mechanism to reduce communication costs and average variance of aggregated data,we additionally integrate gradient quantization with local differential privacy in the algorithm.we also include adaptive optimization on both the server and the client to accelerate aggregation,alleviate over-fitting,and reduce fluctuations caused by noise addition.Experimental results demonstrate that the 1-bit*Ada OP scheme significantly lowers communication overhead compared to other schemes while improving accuracy.(2)To remedy federated learning application scenarios involving multiple communication rounds,we propose an algorithm based on secure shuffle that utilizes the privacy amplification function of the shuffle differential privacy mechanism to realize the impact of local disturbance close to central disturbance.At the same time,we incorporate a client-side parameter accumulation mechanism and the Top-k scheme based on Ada Belief optimization to speed up the aggregation process.Furthermore,we introduce a novel shuffler filling strategy that dynamically selects different filling techniques depending on the quantity of dimensions collected,lowering the amount of injected noise,and improving precision while ensuring privacy amplification.Finally,the feasibility of the scheme is theoretically proved through the privacy gain analysis,and when compared with prior privacy-preserving federated learning approaches,it also demonstrates that this algorithm improves aggregation speed and accuracy,with a negligible reduction of only 1.1% compared to central differential privacy.