The Application Of Adaptive Quasi-newton Acceleration Algorithm In The Composite Function Optimization Problem

With the rapid development of AI in the age of big data,machine learning has become an indispensable part of science and technology.As the core part of machine learning,structural risk minimization involves the solution of composite function optimization.It has become a hot research topic to find optimization algorithm with better performance.In this thesis,an adaptive quasi-newton acceleration scheme is proposed in combination with the adaptive L-BFGS rule in order to further improve the numerical optimization performance.The whole thesis consists of the following six chapters:Chapter 1 introduces the background and research status of composite function optimization,including the first-order optimization and quasi-newton algorithm.Chapter 2 reviews stochastic gradient descent algorithms SAG and SVRG,the related research of BFGS algorithm and its variant L-BFGS,as well as other quasinewton algorithms,introduces the update rules of the adaptive quasi-newton algorithm in detail.Chapter 3 mainly introduces the definition and properties of Moreau envelope,the approximate point algorithm and its variant.Chapter 4 introduces the detailed steps and parameters of QNing,including its rate of convergence and global complexity.Combining QNing with the modified adaptive quasi-newton method,the effect of acceleration scheme on the performance of QNing is discussed.In chapter 5,numerical experiments are carried out on the machine learning optimization model with different regularization.The adaptive QNing algorithm is evaluated by using different data sets.Experiments show that the scheme can achieve faster convergence than other schemes in some cases,and the performance can be improved by proper regularization parameters and smoothing coefficient.Chapter 6 summarizes the work of this thesis,and makes a reasonable evaluation on the advantages and disadvantages of adaptive acceleration scheme,considering the corresponding improvement methods and the prospect of the application.