Research On Sketch-Based Image Retrieval Using Metric Learning Methods

With the development of the Internet and information technology,the amount of image data generated by the Internet is increasing day by day,which challenges the effectiveness of image retrieval methods.Text-based image retrieval methods are affected by semantic ambiguity and the lack of image annotations,while contentbased retrieval methods require accurate query images matching the query intention well,which are not always easy to find for users.Sketch-based image retrieval takes hand-drawn sketches as query,allowing users to draw and find related natural images and alleviating the need of submitting a query image.However,there is a huge domain gap between sketches and natural images.Many feature descriptors,such as texture,are not available in sketches.It is necessary to map the features of both domains to a common feature space for retrieval.To this end,this thesis proposes algorithms for sketch-based image retrieval.This thesis introduces the backgrounds and status of sketch-based image retrieval and sketch recognition,discusses the related theories of metric learning,hashing,and deep neural networks,and then proposes a sketch-based image retrieval algorithm based on spatial co-attention network.Then this thesis describes a semi-supervised retrieval method with re-ranking to cope with the scarcity of labeled data.The contributions of this thesis are as follows:First of all,we introduce the edge map as an intermediate modality to reduce the domain gap between natural images and sketches.Based on the assumption that the key regions of images and the corresponding edge maps should be close,we propose a deep learning method that uses cross-domain spatial co-attention to obtain attention mask and feature fusion.For the requirement of compact intra-class distribution in cross-domain retrieval,the intra-class distance loss is added to the original triple loss.In addition,we also design an auxiliary classification task to retain more semantic information in the coding results and provide a quantization method to produce binary codes suitable for higher efficiency in large-scale retrieval.Secondly,to overcome the lack of labeled data,this thesis studies the sketchbased image retrieval algorithm in the semi-supervised setting.The algorithm obtains pseudo-labels of unlabeled data through a pre-trained classification model,and then samples unlabeled images according to the confidence of the classification.Parameters of the retrieval model is updated using both labeled data and sampled unlabeled data,and it can improve the retrieval performance when there lacks sufficient labelled data.Besides,the semantic information of Top k retrieval results is used to re-weight the distance between the query sketch and images in the gallery based on the fact that Top k retrieval results show higher correlation with the query.In this way,the retrieval results are reranked to further improve the performance of proposed model without introducing extra parameters.