| Multimodal knowledge graph is a knowledge graph constructed based on cross-modal information from multiple heterogeneous sources.By integrating different modal data,multimodal knowledge graph contains richer semantics and has deeper reasoning capabilities.Traditional knowledge graph construction only requires focusing on the knowledge representation of a single modality.However,when constructing a multimodal knowledge graph,challenges such as entity linking between different modalities arise.Additionally,how to find enough high-quality images for entities and how to select representative images for visual concepts are also important issues in constructing a multimodal knowledge graph.In response to these problems and challenges,this thesis studies the construction method of multi-modal knowledge graph,proposes a framework for constructing multi-modal knowledge graph based on contrastive learning,and designs and implements a multimodal knowledge graph system.The paper’s work mainly includes the following three parts:(1)A multimodal knowledge graph construction method based on contrastive learning is proposed.This thesis constructs a text knowledge subgraph and an image knowledge base respectively,and proposes a cross-modal linking module to fuse the knowledge of the two.Finally,a reordering module based on bipartite graphs is proposed,where the text and image entities are treated as two node sets of a bipartite graph.The reordering problem is treated as a maximum weight perfect matching problem to rank the search results and improve the relevance and quality of cross-modal searches.Ultimately,a multimodal knowledge graph is successfully constructed.(2)Design and implement a multimodal knowledge graph system.Through the analysis of business requirements,this thesis determines the role permissions and functional requirements of the system to ensure the accuracy and usability of the business.At the same time,this thesis introduces the overall architecture design of the system,clarifies the functional module division of the system,and expounds the design idea and implementation method of the database in the system.Finally,the thesis introduces the detailed design and realization of the main functional modules in the system.(3)The effectiveness of the multi-modal knowledge graph construction method based on contrastive learning proposed in this thesis is proved by experiments,and the system is tested to prove its usability.The paper conducts experiments on the data set constructed in this thesis and the existing multimodal data set,and analyzes the results.In the comparative experiments and analysis,it is verified that the data set constructed by the multi-modal knowledge graph construction framework proposed in this thesis has good performance improvement on the link prediction results in MR,R@1,R@5,and R@10 indicators.In addition,this thesis tests the main functional modules of the system,describes the test cases and actual results,and the test results meet expectations. |
