| With the rapid development of cloud computing,Internet of things and multimedia technology,while meeting the needs of traditional multimedia services,such as audio and video,people begin to pursue haptic sensory experience.Multi-modal services integrating audio,video and haptic information have been widely regarded as one of the killer applications in the era of Beyond Fifth Generation(B5G)in mobile communication system.Therefore,technologies for cross-modal communications have emerged.However,there are many problems in the process of cross-modal communications.Firstly,during the process of multi-modal data communication and transmission,wireless channel noise pollution and data loss may be encountered,which seriously affect the quality of cross-modal communications.Secondly,different from the traditional transmission scheme of directly transmitting images rather than features in the wireless channel,during the process of cross-modal communications,the problem of limited wireless channel bandwidth,which limits the transmission of highly compressed images,may be encountered.Thirdly,multi-modal data may encounter information security problems caused by data privacy disclosure during transmission,which threaten the security of cross-modal communications.Based on this,the key technologies of information recovery method for cross-modal communications are studied in this paper.The research work of this paper is mainly reflected in the following three aspects:First,this paper studies the information recovery method based on wireless edge for cross-modal communications.Firstly,a cross-modal communication system framework for information recovery based on wireless edge is proposed.The audio,video and haptic stream data collected at the transmitting end are imaged and transmitted in the channel with wireless noise.The information recovery is realized by using the data retrieved from the existing database at the edge node of the receiving end.Then,a cross-modal retrieval method for audio,video and haptic streams is designed,which semantically associates the heterogeneous semantics of different modes,and the retrieval method is used to realize information recovery.Finally,experiments are carried out on the public material surface multimodal dataset and the data collected by the actual cross-modal communication platform.The experimental results show that this method has better information recovery effect than the traditional machine learning and deep learning methods.Second,this paper studies the information recovery method based on joint source and channel coding for cross-modal communications.Firstly,a cross-modal communication system framework for information recovery based on joint source and channel coding is proposed.The audio,video and haptic stream data collected at the transmitting end are encoded and transmitted after image preprocessing,and then decoded after passing through the channel.The information recovery is realized by using the data retrieved from the existing database at the edge node of the receiving end.Then,a cross-modal retrieval method based on joint source and channel coding for audio,video and haptic streams is designed,which semantically correlates the heterogeneous semantics of different modes,and the retrieval method is used to realize information recovery.Finally,experiments are carried out on the public material surface multimodal dataset and the data collected by the actual cross-modal communication platform.The experimental results show that this method has better information recovery effect than the method based on wireless edge.Third,this paper studies the encryption domain information recovery method based on hyperchaotic pseudo-random sequence for cross-modal communications.Firstly,a cross-modal communication system framework for information recovery in encryption domain is proposed,which encrypts the audio,video and haptic stream data collected at the transmitting end after image preprocessing,and then the ciphertext information is transmitted in the channel.The ciphertext data retrieved from the existing database at the edge node of the receiving end is used and decrypted to realize information recovery.Then,the specific encryption,decryption process and the cross-modal retrieval method in the encryption domain are designed,so that the heterogeneous semantics of different modal ciphertext data are semantically related,and the information recovery is realized by retrieval.Finally,experiments are carried out on the encrypted ciphertext data of public material surface multimodal dataset and data collected by the actual cross-modal communication platform.The experimental results show that this method has better information recovery effect than traditional image encryption method based on hyperchaotic pseudo-random sequence. |
