Byzantine-Robust Distributed Optimization Based On Model Aggregation

The rapid development of new technologies in recent years has led to an unprece-dent growth of data collection.A single machine can not deal with such large machine learning problems with tremendous data.On the other hand,a central solution is not even an option when data are inherently distributed.In order to make these types of datasets accessible as training data for machine learning problems,many distributed machine learning framerworks have been proposed.The workloads are partitioned into worker machines,which access the globally shared model as they simultaneously per-form local computations to refine the model.However,distributing workloads over several machines induce a higher risk of failures,including data corruptions,communi-cation failures or malicious attacks,which can bias the model learning process.So,we focus on the robust distributed optimization problem under Byzantine fault model.We consider a genneral distributed system,consisting a master and m workers,among which a fraction of workers are Byzantine that may send arbitrary incorrect messages to the master due to data corruptons,communication failures or malicous attacks,and consequently bias the learned model.In this thesis,we propose a class of robust stochastic subgradient methods for distributed learning from heterogeneous datasets at presence of an unknown number of Byzantine workers.The key to the pro-posed methods is a regularization term incorporated with the objective function so as to robustify the learning task and mitigate the negative effects of Byzantine attacks.The resultant subgradient-based algorithms are termed Byzantine-Robust Stochastic Aggre-gation methods,justifying our acronym RSA used henceforth.In contrast to most of the existing algorithms,RSA does not rely on the assumption that the data are independent and identically distributed(i.i.d.)on the workers,and hence fits for a wider class of applications.Theoretically,we show that:·RSA converges to a near-optimal solution with the learning error dependent on the number of Byzantine workers;·The convergence rate of RSA under Byzantine attacks is the same as that of the stochastic gradient descent method,which is free of Byzantine attacks.Numerically,experiments on real dataset corroborate the competitive performance of RSA and a complexity reduction compared to the state-of-the-art alternatives.