Research On Distributed Training Algorithm Oriented Deep Learning

In recent years,according to the development of the 5G-enabled Internet of Things,the number of wireless terminals and the data they generate has shown explosive growth tread.Computation intensive deep learning task have posed serious challenges to centralized training of deep modes in terms of computing resource and communication connection.Hence,the computing resource need to be deployed in close proximity to data source,and deep models are trained in a distributed way to alleviate the need for computing and communication resources.In order to address this issue,Alternating Direction Method of Multipliers(ADMM)is used to train the computation-intensive,distributed and privacy deep model by exchange of parameters instead of data and parallel computing mode.Thus,the ability of handling distributed data can be enhanced.This thesis studies the distributed classification algorithm based on deep network and ADMM.The procedure of the algorithm can be described as follows: first,each node trains its own deep nerwrok by its local data independently;Secondly,the above trained deep model can be used to generate the features for its local data.On this basis,ADMM is implemented to optimize the global classification parameters,and the back propagation method is executed to update the paramters of all feature layers of its own deep networks;Finally,the effectiveness of the algorithm is evaluated on the CIFAR-10 dataset.However,the above distributed classification algorithm is not implemented in an end to end manner.This thesis proposes a distributed classification algorithm based on end-to-end deep learning.A batch ramdomly sampled from the training set is used to train deep model,a twice-forward scheme is presented to incorporate with ADMM and back propagation,which can update the network parameters after each mini-batch.The procedure of our proposed method involves the following steps.First,in each node,a mini-batch of traing data is used to train the deep network via forward propagation,and the feature of input data can also be obtained by the output of various feature layer.Second,ADMM is used to optimize the the full connected layers of all nodes and update all the full connected layer parameters.Furthermore,in each node,the above mini-batch is forwarded for the second time through the deep network whose the full connected layer parameters is updated in the previous step.Finally,the connected layer parameters are fiexed,the back propagation is used to update all parameters of all feature layers.We validate the effectiveness of our proposed method on CIFAR-10 dataset.