| Optimization problem is one of the most important research directions in computational mathematics.In the field of deep learning,the choice of optimization algorithm is also a top priority of the model.Even when the data set and model architecture are exactly the same,adopting different optimization algorithms is likely to lead to radically different training effects.Stochastic gradient descent(SGD)algorithm is a very common optimiza-tion algorithm in neural network model training.However,the high variance oscillation of SGD algorithm makes it difficult for the network to converge stably.Quasi-hyperbolic momentum algorithm(QHM)is a simple alteration of momentum-based SGD.Its update rule can be regarded as a weighted average of momentum and SGD’s update rule.This algorithm has a good effect on reducing the variance.However,the convergence of QHM algorithm has only been analyzed under strongly convex condition.in practical application,the objective function is often non-convex,so the convergence analysis of QHM algorithm in the case of non-convex has its theoretical value and practical significance.It is the idea of ODE method to approximate a discrete time stochastic system by a deterministic continuous stochastic system.Based on the ODE method,this paper presents the convergence analysis of the discrete time stochastic optimization algorithms SGD and QHM under the condition that the objective function is differentiable and non-convex.Firstly,we introduce a continuous-time version of SGD in the form of an ordinary differ-ential equation.Secondly,we prove the existence and uniqueness of the solution of the ordinary differential equation and the convergence of the solution to the stationary points of the objective function.Thirdly,the interpolated process obtained by SGD iteration is determined to be weakly convergent to the solution of the corresponding ordinary differen-tial equation.Finally,the convergence in the long run of SGD iteration to the stationary points of the objective function is established.For QHM algorithm,we go through the same steps for convergence analysis.Similar to SGD algorithm,a continuous-time version of QHM algorithm is introduced and obtains a more complex ordinary differential equation.We establish the existence and uniqueness of the solution of the equation and the convergence to the stationary points of the objective function.On the basis of determining that the interpolation process of QHM iteration weakly converges to the solution of its ordinary differential equation,the convergence of QHM iteration to the stationary points of the objective function is established. |
