Research On Lightweight Technology Of Deep Neural Network For Image Recognition

In recent years,artificial intelligence has set off a revolution upsurge in many fields.Deep neural network model has achieved great success,and has been widely used in image classification,object detection and other computer vision tasks,which is of great significance to the realization of real artificial intelligence.However,due to the limitation of storage space and power consumption,the storage and calculation of neural network model in embedded devices is still a huge challenge.The lightweight technology of deep neural network model is to solve this problem.By optimizing the training method of the network,the structure of the model or the storage mode of the parameters,the accuracy of the original model can be maintained as much as possible while requiring less storage space and computing resource.This paper mainly focuses on the knowledge distillation method in deep model compression,and proposes a novel knowledge distillation method based on spatial and channel-wise hybrid attention transfer method,which is used to better extract the knowledge information in complex teacher neural network and guide the training process of student network.In order to transfer knowledge from teacher to a different-width student,we further propose a channel-wise selection method to select part of the most active channels of teacher network to guide student,which is actually a further attention to the channel-wise information:we pay more attention to the active channels and give up those channels that are not active enough.Experiments on four public datasets demonstrate the effectiveness of our approach in improving the training outcomes of student networks,and the hybrid attention transfer achieves state-of-the-art performance on knowledge distillation tasks.