| Entity resolution is one of the 19 research topics in the field of data quality.Imagetext cross-modal entity resolution aims to find image and text with the same semantics.However,image-text cross-modal data has the characteristics of heterogeneous underlying data features and semantic correlation of high-level features,namely "heterogeneous gap" and "semantic gap".To solve the above problems,this paper uses the visual semantic embedding framework to carry out the following research from four aspects: semantic correction within a modality,pooling strategy,data augmentation,and cross-modal triplet loss function.Due to the excellent nonlinear fitting ability of the deep learning model,aiming at the problem that the existing image-text cross-modal entity resolution model based on visual semantic embedding cannot well construct the fine-grained semantic correction and generate a global representation of data in the mode,a visual semantic embedding with graph reasoning and pooling is proposed.The graph convolution neural network is used to construct the fine-grained semantic correction between image and text data modes,and the unified pooling operation is used to aggregate the fine-grained feature map to generate a global vector representation for cross-modal data semantic alignment.It provides an improved model for image text cross-modal entity resolution.Based on the previous work,aiming at the problem of introducing new training parameters and ignoring the correction of fine-grained feature information,the model is optimized,and a visual semantic embedding method based on softmax pooling is proposed.The softmax pooling operation is introduced into the visual semantic embedding model for the first time.Without introducing new training parameters,it can adaptively calculate the weight between fine-grained eigenvalues,weighted sum to generate a global unified representation,retain the association information between finegrained features,reduce the operation time and improve the performance of the model.However,the existing visual semantic embedding methods rely on a large number of annotation data and do not use data augmentation technology to expand the training data.To solve this problem,a data enhancement method suitable for image-text crossmodal entity resolution is proposed.The text is enhanced by predictive replacement,random deletion,and random replacement;The image data is enhanced by predicting and replacing the fine-grained features of image target recognition.Experiments verify the universality of data augmentation technology,which can effectively reduce the training sample size and improve the performance of the model.Finally,in terms of semantic alignment of cross-modal data,aiming at the slow convergence speed of cross-modal triplet loss with hard negative samples,a unified crossmodal triplet loss function is proposed.By introducing two expansion factors controlling cross-modal triplet and anchor point,we can pay more attention to difficult negative sample triplet and difficult anchor point in the process of model optimization,At the same time,the optimized samples are not ignored,which significantly improves the convergence speed and generalization ability of the model.This paper uses the visual semantic embedding framework to solve the problem of image-text cross-modal entity resolution,carries out research work on the problems existing in semantic correction,pooling strategy,data augmentation,and semantic alignment,and puts forward corresponding solutions respectively,which has important theoretical and application value for solving the problem of image-text cross-modal entity resolution and improving data quality. |
