Research On Copula Variational Inference

Variational inference is a common method for computing posterior distributions,and plays a central role in machine learning because of its fast convergence and solid theoretical foundation.Although this idealized assumption can simplify the optimization process and improve the computational feasibility,it ignores the posterior correlation(posterior dependence)among the hidden variables,which is not applicable to the models with variable correlation.Copula Variational Inference(CVI)is one of the mainstream methods to solve the above problem.By using the Copula function to capture the dependencies between hidden variables in the model,although comprehensive and complete information can be obtained,there are the following problems:(1)Since the Copula function does not take into account the sparsity of the dependencies,thus capturing some non-essential dependencies and reducing the accuracy of the variational inference approximation;(2)the Copula function adopts the full-rank method,which makes the computational complexity the square of the number of hidden variables and increases the computational complexity.In this paper,the CVI method is studied and improved for the above two problems,and the sparse Copula variational inference method is proposed,and then the sparse Copula hierarchical variational inference method is proposed because its acquisition of correlation information after sparse is not comprehensive enough.The specific work is as follows.(1)A Sparse Copula Variational Inference(SCVI)method is proposed to address the problems that the Copula function in CVI ignores the sparsity of hidden variable dependencies and increases the complexity due to the full-rank calculation.The sparsity of the Copula representation is controlled by adding a sparse induced regularization to find a more compact representation and remove unimportant dependencies.This is done by adding L1 parametrization to the Copula parameters to achieve the regularization goal.Experiments are conducted to validate the approximate performance of the SCVI method based on Gaussian mixture models and latent space models on synthetic and real application datasets,comparing the traditional MFVI and the improved CVI-based methods.The experimental results show the ELBO values of the three methods: SCVI>CVI>MFVI,which shows that SCVI not only inherits the ability of CVI to capture the correlation of posterior distribution,but also further improves the approximation accuracy of variational inference by increasing the sparsity.(2)Although SCVI makes up for the shortcomings of CVI,the sparse Copula function leads to less comprehensive acquisition of dependencies among hidden variables.To solve this problem and further improve the approximation performance of SCVI,on the basis of work(1),we combine the Hierarchical Variational Model(HVM)variational prior structure,an improved Sparse Copula Hierarchical Variational Inference(SCHVI)method is proposed.The hierarchical structure of HVM is introduced in the SCVI method,which can enrich the structural relationships of the model and further obtain the hidden variable correlations without increasing the computational complexity while maintaining the conditional independence among the hidden variables.The specific method is to use the sparse Copula function of SCVI as the prior distribution of the HVM variational parameters.Based on the Gaussian mixture model,the performance of the SCHVI method is tested on synthetic and MNIST real datasets,and compared with the MFVI,CVI methods and the SCVI method proposed in the previous work,and the ELBO values of SCHVI are obtained to be significantly better than the other three comparison methods.From the experimental results,it is clear that the hierarchical variational prior structure combined with HVM can better capture the posterior dependencies among the hidden variables and further improve the approximation performance of the SCVI method.