Cross-Domain Few-Shot Image Classification Based On Knowledge Distillation

Motivated by the rapid learning ability of humans,Few-Shot Learning(FSL)aims at learning to recognize novel instances from unseen categories by a limited amount of labeled data.Recently,most FSL work relies on an auxiliary set with a large amount of labeled data.Due to difficulties in data acquisition,many realistic application scenarios utilize labeled data from other domains to train the model to provide prior knowledge.However,it will inevitably result in the problem of domain shift between the auxiliary set and the test set,which severely hurts the performance on the test set.Therefore,improv-ing the generalization ability of the model among different domains is becoming an im-perative challenge for FSL,which is defined as Cross-Domain Few-Shot Learning(CD-FSL).Based on metric learning,this paper proposes two knowledge distillation based model structures to complete the CD-FSL task.Firstly,to address the problem of domain shift between the source domain and the target domain,we construct a model based on multi-teacher collaboration and multi-level knowledge distillation,which utilizes the framework of the teacher-student net-work in knowledge distillation to effectively transfer knowledge,so that the model has a better generalization ability.Inspired by the idea of meta-learning,we introduce an episode-based training mechanism.Through the combination of task-oriented knowl-edge distillation and multi-teacher collaboration,it not only provides rich and effec-tive knowledge to the student network but also ensures that the student network quickly adapts to few-shot tasks.In addition,we exploit multi-level knowledge from the teacher network,respectively extracting the output prediction and sample relationship of the teacher network as supervision information to guide the training of the student network from different angles.It improves the efficiency of knowledge distillation and better transfers knowledge from the source domain to the target domain.Secondly,from the perspective of teacher integration and relationship modeling be-tween samples,this paper puts forward a method based on reweighting and contrastive knowledge distillation.In order to effectively aggregate the knowledge from multiple teacher networks,a task-based reweighting module is introduced to adaptively adjust the importance of each teacher network under the current few-shot tasks.It avoids negative knowledge transfer to a certain extent and alleviates the problem of knowledge interfer-ence caused by multiple source domains.Besides,inspired by the contrastive learning technology,we utilize triples to model the relationship between samples.Based on it,we propose a prototype-guided contrastive knowledge distillation method,which extracts the structural information between samples from the two aspects of query-prototype and prototype-prototype.The structural information is served as the supervision signal of the student network so that it will learn more robust and discriminative feature repre-sentations to achieve better generalization in the target domain.This paper conducts experiments on four benchmark datasets of cross-domain few-shot learning.The experimental results show that the proposed method can effectively improve the classification accuracy and generalization ability of the model.