Distributed Deep Learning Based On Approximate Newton Method

In many machine learning problems,it is often necessary to solve the optimization problem of large-scale models.The optimization problem mainly studies the optimal approaches and schemes of various problems,and plays an increasingly important role in scientific calculation and engineering analysis.As an effective method to solve this problem,approximate Newton-type method is widely used in computer vision,statistical machine learning,biological information and data mining.With the rapid development of deep network,the scale of data and parameters is also increasing.In recent years,although researchers have made great progress in improving GPU hardware performance,improving network architecture and training methods,it's still very difficult to train deep network model with a single machine on large data sets.Therefore,distributed optimization and its training on large-scale deep network has always been a challenge in the field of deep learning research.With the increase of deep network data and model parameters,the traditional single-machine training has the problems of long training time and limited memory capacity,which can be well solved by distributed optimization method.Therefore,the distributed approximate Newton-type algorithm is introduced into the research of distributed neural networks.The algorithm distributes the whole sample to several computers on average,which not only reduces the amount of data needed to be processed by each computer,but also completes the training task efficiently by means of communication between computers.Firstly,the synchronization strategy of distributed approximate Newton-type method is simulated in parallel with MATLAB.In the experiment,the data sets are divided equally and calculated in parallel.Under the same matrix,the running time of parallel optimization is much less than that of serial optimization,and the convergence is also guaranteed.Secondly,the distributed approximate Newton-type algorithm is used to train deep network.The algorithm enables each Worker node in the Parameter server cluster to complete the calculation using equally distributed local data,and communicates with the Server node to obtain the results of synchronous optimization.When using distributed approximate Newton-type method to train the same deep network,as the number of GPUs increased by 2 times,the training time decreased exponentially by nearly 2.This is consistent with the ultimate goal of this paper,that is,on the premise of ensuring the accuracy,using the existing distributed framework to realize the deep network training of the approximate Newton method to improve the operation efficiency.