Neural Network Architecture Optimization Based On Feature Knowledge

Deep Neural Networks(DNNs)have become one of the core technologies of artificial intelligence(AI)development with their powerful representation ability.However,storage issues and inference speed limit the application of neural networks on edge devices.Traditional methods use the attention mechanism or knowledge transfer to improve model performance,ignoring the hidden feature knowledge behind the pre-trained network,which can guide the network structure adjustment.Lightweight neural networks reduce model parameters and improve the inference speed by designing more efficient calculation methods.While current exploration of lightweight network architecture design in improving performance mainly focuses on the reconstruction of the convolution operation itself,and treats the convolution kernel sizes of all layers equally.It ignores the feature scales extracted from different depths and different channels are different.This thesis uses feature knowledge extracted from the pre-trained model to modify the kernel size and to achieve the same-layer and cross-layer multi-scale information fusion.Meanwhile,redundant filters are pruned to improve the model’s characterization ability while compressing the model size to optimize the network structure.The main innovations of this article are as follows:1.This thesis proposes an efficient method for extracting feature knowledge via group convolution.This method takes into account the multi-scale information in the spatial domain,and realizes the extraction of the spatial domain feature knowledge of each channel through group convolution,which can extract the maximum response region of each channel in batches and efficiently,and guide the model to pay attention to the unique visual mode of each channel and improve the model performance.2.This thesis proposes a mix convolution measurement strategy via feature response region.The proposed indicator adjusts the size of the convolution kernel of the lightweight network channel by channel to achieve mix convolution within and across layers which not only improves the performance of the model,but also compresses the model.The proposed method has been verified by a large number of experiments on multiple tasks and datasets,and compared with the current artificially designed convolution module.The experimental results show that the algorithm of optimizing the neural network structure based on feature knowledge proposed in this thesis can not only select the appropriate size of the convolution kernel to improve the accuracy of the model,but also can identify the redundant filter to make the model more compact.Compared with the artificially designed convolution module,the proposed method also achieved better performance.By comparing the results over different tasks,this thesis finds that although the architectures are the same,the networks can adaptively select different mix convolution ratios according to the task,which provides new ideas for the network structure design of different tasks.