| In recent years,deep learning has developed rapidly in the field of computer vision,and efficient network architectures emerge one after another.The deep convolutional neural networks,which use high-order pooling instead of global average pooling to aggregate features,have achieved remarkable results in various visual tasks.However,the effectiveness of high-order pooling on deep convolutional neural networks has not been well studied,which limits the design and development of high-order pooling deep convolutional neural networks.This thesis focuses on the mechanism of high-order pooling on deep convolutional neural network,analyzes the effectiveness of high-order pooling from the perspective of optimization,and proposes a more efficient high-order pooling method.The main work and innovations are as follows:(1)In this thesis,the effects of high-order pooling methods on deep neural networks are firstly explored from the perspective of optimization through empirical comparison and theoretical analysis.At present,the effectiveness of high-order pooling is mainly analyzed from the perspective of statistical modeling or Riemannian geometry,but some properties of high-order pooling deep neural network still lack reasonable explanation,such as accelerated network convergence.From an optimization perspective,we find that high-order pooling in deep convolutional neural networks can optimize the Lipschitz property and gradient predictability of loss landscapes,and is associated with secondorder optimization in the process of back propagation.Based on the above analysis,this thesis further reveals the advantages of high-order pooling in deep neural network optimization,such as accelerated network convergence,improved robustness to distortion samples,and good generalization ability in different visual tasks.(2)This thesis presents a maximum singular value matrix power regularization technique based on partial singular value decomposition.The regularization techniques of high-order pooling(e.g.,matrix logarithmic regularization,matrix power regularization,element power regularization)play a crucial role in the effectiveness of high-order pooling.At present,the most effective deep learning method is matrix power covariance regularization,which uses Newton-Schultz iterative approximate matrix power covariance matrix to accelerate model training,but newton-Schultz iterative method needs several iterations to effectively approximate matrix,which increases the computation amount of network forward propagation and back propagation.The proposed method only needs one iteration to effectively approximate the matrix,and has the advantages of low computation and fast model training compared with similar methods.The theoretical analysis of high-order pooling in this thesis provides an inspiring view for understanding high-order pooling and provides theoretical support for researchers to better explore the advantages of high-order pooling in deep convolutional neural networks.In addition,the maximum singular value matrix power regularization technique proposed in this thesis enables the high-order pooled deep convolutional neural network to achieve competitive results on ImageNet datasets while accelerating the model training speed. |
