Cross-modal Retrieval Technology For Public Safety Monitoring

Multi-modal spatio-temporal data is spatial data with spatio-temporal information,rich semantics,and diverse modalities.With the acceleration of urbanization,urban public safety supervision is becoming more and more important and complex.With the widespread use of the Internet and the rapid development of sensors and information technology,urban public safety monitoring facilities and technical means are more common and diverse;at the same time,the process of urban public safety monitoring will generate a large amount of multi-modal spatio-temporal data.How to realize the multimodal retrieval of multi-modal spatio-temporal data,so as to provide powerful data support for the rapid diagnosis,discovery and early warning of urban public safety events has become an urgent need to solve Issues and research priorities.At present,the research of cross-modal retrieval technology is mainly oriented to the text,video and image data of the computer field,without considering the unique spatio-temporal characteristics of spatial data,and it is difficult to support multi-modal spatio-temporal data cross-modal retrieval that takes into account time,space and semantic information.Therefore,this paper is oriented to the requirements of cross-modal retrieval of urban public safety monitoring,and focuses on the study of cross-modal retrieval technology that takes into account the temporal,spatial and semantic correlation of multi-modal spatio-temporal data.The main research contents of the paper include:(1)Multi-modal spatio-temporal data association model based on semantics.In view of the low-level feature heterogeneity and high-level semantic correlation of multimodal spatio-temporal data,this paper proposes a semantic-based multi-modal spatiotemporal data association model.The model is based on the spatio-temporal data feature-semantic mapping mechanism,and uses the ontology theory to construct a multi-modal spatio-temporal data semantic expression model,which realizes the standardized semantic expression of multi-modal spatio-temporal data;on this basis,this paper proposes The calculation methods of temporal correlation,spatial correlation and semantic correlation realize multi-modal spatio-temporal data correlation construction and correlation measurement,and provide model support for cross-modal retrieval for public safety monitoring;(2)Spatio-temporal semantic integration index method.In this paper,based on the integrated query of time,space and semantics of multi-modal spatio-temporal data,based on Geohash coding and inverted index structure,a spatio-temporal semantic integrated index method is proposed.The index is based on Geohash grid division and coding as a spatial reference,and is divided into coarse-grained and fine-grained secondary divisions according to the time granularity of multi-modal spatio-temporal data.Spatial correlation state inverted structure,semantic inverted structure,and double inverted structure.The index fully considers the spatial distribution,temporal granularity,and semantic association of multi-modal spatio-temporal data,and provides cross-modal retrieval for public safety monitoring.Support for spatial-temporal semantic integrated indexing method that takes into account time,space and semantics;(3)Cross-modal spatial-temporal data retrieval method.In this paper,the multi-modal spatial-temporal data retrieval requirements for public safety monitoring are defined.The multi-modal spatial-temporal data retrieval model is defined.The cross-modal retrieval method of the spatio-temporal change object and the cross-modal change process are proposed.Retrieval method,on this basis,based on the spatio-temporal semantic integrated index proposed in this paper,realize the spatio-temporal relationship query and semantic similarity query of spatio-temporal change objects,as well as the behavior process retrieval and event change process retrieval of the spatio-temporal change process.The monitored cross-modal search provides an effective search method.Finally,in this paper,five typical spatio-temporal data such as trajectory data,geographic video,POI data,vector data and pictures are used as experimental data to instantiate and construct a multi-modal spatio-temporal data semantic expression model,based on the temporal correlation The calculation method of relevance and semantic relevance realizes the relevance construction and relevance analysis of multi-modal spatio-temporal data,and verifies the validity of the multi-modal spatio-temporal data correlation model based on semantics;Integrated index,and compared with the traditional combined index,carried out related experiments and analysis of the cross-modal retrieval of the spatio-temporal change object under different query conditions and the cross-modal retrieval of the spatio-temporal change process.The experimental results verify the effectiveness of the spatial-temporal semantic integration index and the cross-modal spatial-temporal data retrieval method proposed in this paper.