Research On Convergence Properties Of Learning Algorithms For Deep Generative Models

In the era of big data,the digital information that people can obtain from the internet not only grows rapidly in quantity,but also in its forms and diversity.However,such data often lacks effective labeling information,and contains a lot of noise and abnormal data.In this context,deep generative models(DGM)model the joint probabilistic distribution of complex data by utilizing the powerful deep neural networks.It can provide a principled unsupervised learning framework and successfully characterize the uncertainty and noise in data.Because of its good theoretical properties and practical performance,DGMs play an important role in many research fields and industrial products,and have significant theoretical and application value.However,the training algorithms for various DGMs still suffer from convergence issues,like instability,local minimum,or slow convergence speed.These issues make the performance of DGMs largely relying on the proper hyperparameters and sufficient computational resources,which largely impede it from being applied to a new field.Specifically,for generative adversarial nets(GANs),the minimax objective function nat-urally results in an unstable training process? for structured directed probabilistic models(SDPMs),simply optimizing certain divergence between the marginal distributions results in the local minimum with undesirable model behavior? for energy-based models(EBMs),the higher-order derivatives in score matching result in high computational complexity.To generalize DGM to large-scale datasets,this dissertation provides a systematic study of the above challenges by proposing stable,optimal,and efficient training methods for DGMs.In summary,the contribution of this dissertation is three-fold:· We propose a stable training method for GANs based on control theory.We lever-age the widely-adopted Laplace Transform and the closed-loop control method incontrol theory to analyze and stabilize the training dynamic of GANs.Our methodcan successfully improve the stability of various variants of GANs and the con-verged performance empirically.· We propose an amortized structured regularization framework to regularize theSDPMs based on posterior regularization.Our method can use human knowledgeto improve the training of SDPM,and avoid the undesirable local optimal in thetraining process effectively.· We propose an approximation method for higher-order derivatives based on fi-nite difference and apply the method to score matching.Our method is theoret-ically sound and improves the computational complexity of higher-order deriva-tives to linear order.With comparable performance,our method can acceleratescore matching methods from 1.5 to 3 times.