Research On Convolutional Neural Network Compression Algorithm And Application Based On Knowledge Distillation

Convolutional neural networks have emerged as the backbone of various computer vision tasks,but their increasing complexity and parameters have posed challenges for deploying them on resource-limited terminals.Knowledge distillation has shown promise as a model compression technique to reduce the computational burden and improve performance on these devices.However,existing knowledge distillation algorithms still have some drawbacks.In this thesis,we propose novel approaches to improve two different forms of knowledge distillation.The main work contents are as follows:In response to the issue of increased complexity and training time caused by the feature mapping layer in the feature matching-based offline knowledge distillation algorithm,this thesis proposes a sliding window-based feature matching knowledge distillation algorithm.The algorithm discards the feature mapping layer and when the channel dimensions of the feature maps output by the teacher model and the student model are different,the feature map with the smaller number of channels is used as a window to slide and match on the larger feature map,with the stride size equal to the window size.During the sliding process,the mean squared error loss between the feature maps after average pooling is calculated,and this step is repeated until all feature maps are involved in the loss calculation.In addition,the attention score matrix between channels is used for knowledge transfer,and the Softmax regression loss is introduced.This thesis extensively verifies the effectiveness of the proposed algorithm on multiple datasets.On the ImageNet dataset,training the ResNet-18 model with the method proposed in this paper can improve the accuracy by 1.61%compared to the baseline model.In response to the lack of diversity among branches in the existing online knowledge distillation algorithm based on multiple-branch structures.This thesis proposes a MultiArchitecture Peers online knowledge distillation algorithm.It improves diversity among branches by constructing a multi-branch model and proposes a feature learning module to fuse the feature maps outputted by auxiliary branches and obtain a weighted aggregation result.The algorithm adopts a two-stage distillation approach,using the weighted aggregation result to guide the training of the auxiliary branches and then transferring the average prediction result of the auxiliary branches to the target branch.Through extensive experiments,the proposed method performs well on models with different architectures.Specifically,on the ImageNet dataset,training ResNet-34 using the proposed method can improve its accuracy by 1.39%compared to training the target model alone.An embedded image classification system based on knowledge distillation was designed and implemented.Based on the above research results,an embedded image classification system integrating data acquisition,model training,and terminal deployment is designed and implemented.Moreover,the two knowledge distillation algorithms proposed in this thesis are applied to model training,which significantly improves the classification accuracy of the lightweight model,effectively reduces the resource consumption of the embedded terminal,and improves the operation speed.