Multi-Task Bayesian Optimization Based On Gaussian Processes And Conditional Neural Networks

In real life,there are many problems that can not be expressed in concrete form.It is difficult to analyze these problems to provide clues for finding the optimal value.Bayesian optimization is a method suitable for solving such unknown black-box problems,and it can well handle the evaluation of enpensive objective functions(such as long running time and high computational cost).Many optimization methods are based on a single problem,namely singletask learning and optimization,but the related information of other similar tasks is ignored to further study data features.Single-task learning learns from scratch alone,and often encounters problems such as large noise,high data dimensions,and small data volume that have a large impact on the results.Multi-task learning is to combine similar single tasks,which can effectively increase the training samples to eliminate the interference and improve the generalization performance of each single task by sharing learning information between tasks.This paper aims to study multi-task Bayesian optimization.The main work and achievements include the following aspects:(1)We Apply the multi-task Gaussian processes(MTGPs)to the Bayesian framework,MTGPs simultaneously optimizes multiple problems by learning the shared covariance function and the task-correlation covariance matrix based on the input correlation characteristics,and the variance matrix models the similarities between tasks to share the information.The optimization performance of each task is improved finally.(2)The computational complexity of MTGPs increases exponentially when the amount of data increases.Two multi-task conditional neural processes(MTCNPs)models are proposed to solve this problem.MTCNPs does not rely on the mathematical modeling of the taskcorrelation covariance matrix,but through the special neural network structure to construct the similarity.We also apply MTGPs to the Bayesian framework.According to the different training data forms,the multi-task conditional neural network is divided into the one-to-many model and the many-to-many model.The former utilizes information based on input-related characteristics,while the latter mainly utilizes the characteristics of information sharing between tasks,and two different training mechanisms are proposed in the many-to-many model.(3)In the experiments,several complex multimodal functions are simulated into intricate optimization problems in real life,and the multi-task Bayesian optimization models proposed in this paper and the single-task Bayesian optimization models are tested for these problems.Besides,Gaussian noise is added to the functions,and then our algorithms optimize these functions to verify the robustness.The results show that the proposed multi-task Bayesian optimization models are better than the single-task models in approximate optimal values and the convergence speed.(4)With the development of deep learning,the research on hyper-parameter learning has become very urgent.We apply the multi-task models proposed in this paper to hyper-parameter learning of complex networks and test the effectiveness of the multi-task framework by setting different hyperparameter dimensions.The experimental results show that compared with the single-task Bayesian models,the multi-task Bayesian models can find the hyperparameter combinations that make the network more accurate.