Study Of Stochastic Gradient Descent Batch Optimization Based On Information Transmission Maximization Criteria

In high-dimensional space,the optimization algorithms in deep neural networks tend to fall into the saddle point and are not easy to converge to the minimum.Stochastic gradient descent algorithm is one of the common optimization algorithms in deep neural networks and can effectively improve the performance of deep neural networks by injecting random perturbation.Since the reliability of first-order gradients extracted from batch samples is different,it is necessary to optimally allocate the random perturbation of the stochastic gradient.Therefore,we propose a stochastic gradient optimization based on information transmission maximization method.At first,this paper analyzes the stochastic gradient descent method in deep neural network from the perspective of dynamical system,establishes Langevin equation,and obtains the stochastic gradient descent description and optimization based on Fokker-Planck equation,and establish the relationship between random signal power adjustment and the algorithm escapes from the saddle point.Then,based on the information transmission maximization criterion to balance the random perturbation of stochastic gradient and realize the optimal allocation of the random perturbation,use water-filling power allocation to increase attention to low-power samples or features and reduce attention to high-power samples or features,thereby improves the optimization performance.Image classification experiments are conducted on three deep neural networks and the experimental results indicate that the proposed method can improve the classification performance effectively.