Research On Feature Generation Of Deep Learning Based On Latent Variable Model

Deep learning is a research hotspot of machine learning technology.It can learn different levels of feature information in a large amount of data by multi-layer neural network structure,so as to solve the problems effectively of classification,regression,etc.However,we have to use small datasets to train a classification or a regression model in some data-shortage application.The model may produce serious overfitting,if we train it on small dataset directly.Aiming at this situation,firstly,Feature Generalization Algorithm based on latent variable model is proposed to overcome the problem of small-scale dataset in the training stage.Next,two network optimization methods based on Feature Generalization Layer are proposed to improve the training efficiency and the classification accuracy of the model.The essence of the Feature Generalization Algorithm lies in the construction of the generation model and the definition of the optimization target.This paper structures a layer which modeled by latent variable model,and a pair of parameterized mappings bidirectionally associate the data space with the latent space.The dual objective functions are defined to optimize parameters of the network collaboratively,one of them minimizes the generation error,and the other minimizes the classification error.Evaluations on the MNIST dataset and Chars74 k dataset show that the DNN model when adding with Feature Generalization Layer produces better results than before.Classification accuracy tested by different quantity of training samples increase respectively in the range of 0.42 to 33.33 percent on MNIST Dataset,2.25 to 15.5 percent on Chars74 k nature image Dataset,and 2.5 to 21.25 percent on Chars74 k computer synthetic character Dataset.Experiment results illustrate that the algorithm can enhance the generalization ability of the deep network,and reduce the over-fit phenomena caused by training on small dataset.Compared with other generation models used for data augmentation,Feature Generalization Algorithm simplifies the complexity of the network and improve the efficiency of training.Parallelly Multi-branch Forward Propagation optimization method aims at improving the efficiency and the convergence of model training.Considering the data extracted from the feature extraction layer in network is a matrix composed of multiple feature maps,the proposed method adopts the idea of data distributary processing to establish an independent generalization branch for each feature map and reduce the data complexity.The generation objective function of the network is calculated by the weighted average method of multiple branches,which can reflects the overall performance of Feature Generalization Layer more evenly.Dual Channels Adaptive rectification optimization method is designed to improve the classification accuracy of the model.Through error measurement,we can adaptively rectify feature maps in the generalization channel layer-by-layer,and control the generation error included in the subsequent feature extraction layer's output in a reasonable range.Finally,all of measuring errors are added to the objective function of the network as a constraint term,so that the generated feature graph is more in line with the abstract attributes of the original data.Experimental evaluation of two optimization methods on MNIST Dataset show that: The classification accuracy of the model which optimized by Dual Channels Adaptive Rectification method can maximal increase 4.09 percent;The classification accuracy of the model which optimized by Parallelly Multi-branch Forward Propagation method can maximal increase 1.67 percent,but this method only takes 45%(25)65% of former's training time,training efficiency is higher.Qualitative and quantitative simulation results verify the validity and applicability of the two optimization methods in this paper.