Research On Model Distillation Method Based On Channel Knowledge

In recent years,deep neural networks have achieved great success in visual recognition task,but also brought some application and deployment problems.These huge deep models usually consume a lot of computing and memory resources not only in the training stages but also in the inference stages,which makes it very difficult to deploy deep learning models on resource-constrained mobile devices or embedded systems.Therefore,Knowledge Distillation(KD)as a model compression technique,has attracted more and more attention in the deep learning community.It can not only extract knowledge from a large teacher neural network and transfer it to a small student network,but also make the student network have better performance.However,there are also some problems of ignoring the knowledge transferred from channels in the existing knowledge distillation methods.This thesis focuses on the neural network channel feature knowledge and model distillation methods,and designs and implements a face recognition prototype system based on model distillation.This thesis has done the following three aspects of research work:(1)In order to make full use of the channel feature knowledge between sample instances and reduce the influence of the wrong prediction knowledge output by the teacher network on student learning,this thesis proposes a Channel Correlation-Based Selective Knowledge Distillation(CCSKD)method.In CCSKD,feature-based knowledge is derived from the channel relationships between sample instances.In order to obtain correct response-based knowledge,CCSKD method gradually reduces supervision of logits outputs that contain wrong sample predictions.Compared with other existing KD methods,the CCSKD method can use both the channel correlationbased feature knowledge and the selective response-based knowledge in the process of knowledge distillation.In this thesis,sufficient experiments are carried out on multiple image classification datasets.The experimental results show that CCSKD method can obtain better accuracy in image classification task.(2)In order to make full use of the various knowledge output by teacher network and improve the ability of student network to learn knowledge,Multi-Stage Knowledge Distillation via Student Self-Reflection(MSKD-SSR)is proposed.Existing KD methods mainly focus on improving the efficiency of knowledge distillation from the perspective of teacher network,while ignoring the improvement of distillation efficiency from the perspective of student network.The MSKD-SSR method considers how to effectively allow student network to improve their learning efficiency while considering the teacher network’s transfer of knowledge to the student network.The MSKD-SSR method divides teachers and students into several learning stages.In each stage,teacher network transfers its own channel feature-based knowledge and response-based knowledge to the student network.The student network can learn the knowledge learned in the previous stages by reviewing.In this paper,comparative experiments and ablation experiments are carried out on multiple image datasets.The experimental results verify the excellent performance of MSKD-SSR method.(3)According to the CCSKD method and MSKD-SSR method proposed in this thesis,a face recognition prototype system based on model distillation is designed and implemented.The system mainly includes two modules: system management and face recognition.The system is developed by C++ and Python languages,and selects My SQL database for data read and write operations.The human-computer interface of the system is simple and easy to operate.After many runs and tests,the effectiveness of the two methods proposed in this thesis is verified.