A Novel Multi-modal Integration And Propagation Model For Cross-Media Information Retrieval

Multimedia information is not just a simple combination of text, images, audio and video and other multimedia information, but the interaction and fusion of the various types of multimedia information. Multi-media information is the synthesis of various kinds of multi-modal information, these different types of multimedia data expressed a variety of semantics, and appear in the form of interrelated and mutually complementary. Therefore, it is necessary to research a reliable algorithm for cross-modal retrieval.The main difficulty of cross-modal retrieval is the gap between the low-level features and high-level semantics, and the weak correlations between different modalities. In this paper, we present a novel Probabilistic Latent Semantic Analysis-based (PLSA-based) aspect model and turn cross-media retrieval into two parts of multi-modal integration and correlation propagation. We first use multivariate Gaussian distributions to model continuous quantity in PLSA, avoiding information loss between feature-instance versus real-world matching. Multi-modal correlations are learned in an asymmetrical manner, giving a better control of the respective influence of each modality in the latent space. Then we propose a new propagation pattern to refine multi-modal correlations by efficiently taking the complementary from multi-modalities. Experimental results demonstrate that our method is accurate and robust for cross-media information retrieval.