A Study Of Kernel Mapping Structure And Cross-connected Structure Using Deep Neural Networks

After more than ten years of rapid development,deep learning has gained more and more attention.Deep learning refers to the various methods of training deep neural networks.Based on these methods,deep neural networks can learn suitable and effective features from the massive and complex data,and obtain excellent results in solving practical problems.This is the fundamental reason why deep learning is popular in both academia and industry.The main work of this paper consists of two basic aspects: the deep kernel mapping structure and the deep cross-connected structure.The deep kernel mapping structure aims at solving the kernel learning problem of support vector machines using deep neural networks,and deep cross-connected structure is designed to improve the learning performance of deep neural networks by using shortcut connections.Theoretically,the deep kernel mapping structure can approximate any kernel mapping by a deep neural network,unlike the traditional method that does kernel learning by combining basic kernels with a linear or nonlinear manner.It is expected to overcome the fundamental difficulity of kernel learning.Deep cross-connected structures can integrate different scale features by the concatenating or fusing them.It significantly improves the performance of deep neural networks.In fact,the deep residual network and the densely-connected network,both with shortcut connections,got the best paper in CVPR 2016 and CVPR 2017,respectively.This indicates that the importance of cross-connected structures has been recognized in the field of deep learning.The main innovation research of this paper are summarized as follows:1.Deep neural mapping support vector machine for overcoming the diffi-culty of kernel learningWith the idea of using deep neural networks approximating kernel mapping,the model maps the original input into the feature space with a deep neural network.Different from the implicit function induced by the traditional kernel functions,this model is a new study of universal support vector machine.Moreover,this model combines two stage learning of contrast divergence and gradient descent to jointly train adaptive kernel mapping without kernel techniques.Experimental results show that the deep neural mapping support vector machine is better than the neural support vector machine and radial basis function support vector machine,demonstrating its effectiveness.2.Cross-connected convolutional neural network for concatenating two-scale features and improving learning performanceThe cross-connected convolutional neural network is an eight-layer network architecture,including the input layer,six hidden layers composed of three convolutional layers alternating three pooling layers,one fully-connected layer and the output layer.In this network,the second pooling layer is allowed to directly connect the fully-connected layer by crossing two hidden layers.Experiments on gender classification with ten human face datasets show that the accuracies of the cross-connected neural networks are not lower than that of the traditional convolutional neural networks.3.Concatenating framework of shortcut convolutional neural networks for concatenating multi-scale features and improving learning performanceBy expressing different concatenating styles with a binary shortcut indicator,this framework connects different convolutional layers and pooling layers to the fully-connected layer by shortcut connections with fixed weights,and then lets the fully-connected layer connect to the output layer.We have showed the performance of this framework on different datasets.They are the AR,FERET,FaceScrub and CelebA datasets for gender classification,the CUReT dataset for texture classification,the MNIST dataset for digit recognition and the CIFAR-10 dataset for object recognition.Experimental results show that the shortcut convolutional neural networks can achieve better results than the traditional ones on these datasets,with more stability in different settings of pooling schemes,activation functions,optimizations,initializations,kernel numbers and kernel sizes.Morevoer,shortcut convolutional neural networks are comparable with ResNets and GoogLeNet,as well as outperforming DenseNets,Multi-scale CNN and DeepID.4.Fusing framework of shortcut convolutional neural networks for fusing multi-scale features and improving learning performanceWith the idea of weighted shortcut connections,this framework connects different convolutional layers and pooling layers to a fully-connected layer of fixed units with learnable shortcut connections,and then lets the fully-connected layer directly connect to the output layer.We have showed the performance of this framework on different datasets.They are the AR,FERET,FaceScrub and CelebA datasets for gender classification,the CUReT dataset for texture classification,the MNIST dataset for digit recognition and the CIFAR-10 dataset for object recognition.Experimental results show that the shortcut convolutional neural networks can achieve better results than the traditional ones on these datasets.Additionally,they are more stable in different settings of pooling schemes,activation functions,initializations and occlusions.Morevoer,shortcut convolutional neural networks are comparable with ResNets and can outperform GoogLeNet,DenseNets,Multi-scale CNN,DeepID and CS-CNN.