Research On Algorithms Of Restricted Boltzmann Machine And Their Applications

Deep learning has become a hotspot research in the field of artificial intelligence.Restricted Boltzmann machine(RBM)is one of the deep learning models,which can learn the probability distribution of the input data by drawing the data image into one-dimensional vector.It is widely used in deep neural network because of its strong representation ability and good generation model.The training of RBM will directly affect the performance of the whole depth network.Therefore,how to improve the RBM's algorithm to enhance the network generalization ability and robustness is an important problem for the application of the network.This thesis mainly focuses on the problems of the algorithm performance,the limited momentum acceleration and the limited generalization ability of RBM in the gradient approximation algorithms,momentum algorithms,and the regularization methods,and their related applications in image recognition and seismic data denoising.The main works are summarized as follows:(1)An improved algorithm of product of experts system based on restricted Boltzmann machine is proposed.The product of experts theory is combined with the RBM algorithm and the parameter updating way is all adopted the probability value,which leads to the undesirable recognition effect and slightly worse density models,so we modify the updating rule of the hidden bias parameter to overcome the density problem and obtain better performance given the same running time.An improved algorithm with momentum terms in different combinations is used not only in the RBM pre-training phase but also in the fine-tuning stage for both classification accuracy enhancement and training time decreasing.Through the image recognition experiments on the MNIST database,Extended Yale B and CMU-PIE face databases,and the application of seismic data denoising,the result shows that the proposed algorithm reduces the training time,and improves the efficiency of hyper-parameters optimization,and then the deep belief network can achieve better classification performance in image recognition,and can effectively remove the random noise in seismic data,the new method is effective.(2)We propose an effective algorithm of restricted Boltzmann machine based on momentum method.The improved momentum method is used not only in gradient ascent algorithm but also in gradient descent algorithm for both classification accuracy enhancement and training time decreasing.According to the characteristics of the gradient ascent algorithm,a rapidly ascending momentum method is used in the RBM pre-training phase,which greatly improves the speed of learning.According to the characteristics of the gradient descent algorithm,an improved slowly descending momentum term is also used in the fine-tuning stage to accurately find the best point.Through the recognition experiments on the MNIST dataset,Extended Yale B and CMU-PIE face datasets,the achieved results show that the improved momentum algorithm can effectively enhance the ability of image feature expression and improve the performance.(3)The algorithm based on modified momentum using restricted Boltzmann machine is proposed.Focusing on the gradient approximation algorithm insensitivity to the momentum acceleration and recognition effectiveness in RBM,we modify the updating rule of the hidden bias to avoid the problems of the undesirable recognition performance and limited momentum acceleration.In the stochastic gradient ascending(SGA)and the stochastic gradient descending(SGD)algorithms,two different forms of momentum methods are combined with SGA and SGD to accelerate the convergence of the whole network and improve the classification effect by Gibbs sampling.Recognition experiments on MNIST,Extended Yale B and CMU-PIE databases illustrate the effectiveness and advantages of our proposed method in terms of the computational efficiency and the generalization ability of networks.(4)We propose the improved average contrastive divergence(ACD)algorithm with weight-decay momentum to achieve good performance.Contrastive divergence(CD)algorithm has been proven to be a biased estimate of the true log-likelihood gradient.In order to obtain unbiased estimates of the gradient,we approximate the second term in the log-likelihood gradient by the average of a batch of samples for the RBM distribution using Gibbs sampling.However,not all RBM's gradient approximation algorithms combining with the momentum method can accelerate theconvergence of the network.Therefore,we use the modified momentum algorithm with the weight-decay to approximate the log-likelihood gradient based on the previous research work to obtain better performance.Finally,the proposed algorithm is evaluated on the MNIST database,Extended Yale B,CMU-PIE face databases,and the application of seismic data denoising.The results show that the proposed learning algorithm is a better approximation of the log-likelihood gradient and outperforms the other algorithms,and can effectively remove the random noise in seismic data.(5)We propose the improved log-sum regularization algorithm with the improved gradient approximation and momentum to achieve good performance.Restricted Boltzmann Machine is a strong representation and generative model for unsupervised feature extraction in deep learning.Inspired by the log-sum norm providing more effective sparsity inducing properties in compressed sensing and based on the previous four research work,we add the log-sum regular term to the RBM's objective function and combine the improved gradient approximation method with different momentum terms to improve the representation ability.By adding a log-sum term to the likelihood function of an RBM,the sparsity of each hidden unit can be adaptively learnt to get sparse representation.The different momentum terms are used to enhance the RBM's computational efficiency.The improved gradient approximation algorithm with the log-sum regularization and momentum is a better approximation of the log-likelihood gradient than the traditional CD algorithm.The recognition results on MNIST handwriting,Extended Yale B and CMU-PIE face databases show that the proposed gradient algorithm combines the log-sum regularization with the modified momentum method to improve the sparse feature learning performance and feature expression ability of the RBM network,which show the strong generalization performance and robustness,and then improves the classification effect.The algorithms not only extend the application fields of RBM,but also provide the new research ideas and references for the application methods of deep learning.