Research On The Optimization Strategy Of Model Parameters In Distributed Deep Learning

In the training process of Distributed Deep learning(DDL),Parameter Server(PS)distributes parameters to the work node.After the calculation process,work nodes feedback the result to the server for parameter optimization.However,the traditional DDL system did not take into account the single-layer structure of the Deep Learning(DL)model and the parameter changes on the work nodes,which resulted in the network congestion and the decrease of optimization performance in the heterogeneous environment.Therefore,this paper first proposes an optimization strategy for model parameters to solve the above problems,and then deploys the strategy on PS to verify the performance.Meanwhile,by improving the traditional Stochastic Gradient Descent(SGD)algorithm,a Valuestaleness-aware Gradient Descent(VSGD)algorithm based on model numerical delay is proposed to improve the performance of PS models in heterogeneous environments.The main contributions of this paper are as follows:(1)After the consideration of time consumption and data transmission between Convolutional Neural Network(CNN)and Fully Connected Neural Network(FNN)in the DL model,this paper proposes an optimization strategy oriented to model parameters to improve the utiliazaion of network and the performance of DDL system.According to the characteristics of each layer of DDL model,this strategy adopts strategies to deal with the computing and transmission.Therefore,the utiliazaion of network is improved and the network congestion is alleviated.In this paper,the parameter optimization strategy is applied to PS architecture,and the results show that it can significantly improve the training speed in DDL.(2)To solve the impact of heterogeneity,VSGD algorithm is proposed by analyzing the DL process.Specifically,the algorithm transmits part of data to the server to compare the delay of model values.Therefore,the influence of the calculation results on each work node is adjusted,which improves the performance of PS architecture in heterogeneous environment.This paper develops a DDL system based on Tensor Flow to implement the above algorithm,and evaluates the system performance in homogenerous and heterogeneous environments.The results of experiment show that the proposed architecture and method can achieve better performance.