Multivariate Sensing Data Matching And Applications

With the booming development of Internet of Things(IoT)and Artificial Intelligence(AI)related technologies in recent years,IoT devices with various levels of sensing capabilities and corresponding intelligent applications are greatly changing people’s lifestyles.It can be said that we are now living in a world of multivariate data,so how to make use of the huge amount of multivariate data to provide more intelligent services has become an important issue for every walk of life.The advantage of multivariate data is that it can be used to perceive the world around us in a multi-perspective,multidimensional or multi-modal way.The complementary nature of multivariate data allows for a more comprehensive and accurate picture of events,making multivariate data based sensing applications more reliable and usable.The heterogeneity of multivariate data offers the possibility of more intelligent applications,but also poses challenges in the design of multivariate data alignment and matching,and fusion.This dissertation addresses three scientific problems of multi-source data matching,including multi-source sensing data entity matching,multimodal sensing data entity and semantic matching,and multisemantic text and image matching.Based on this,the method is designed for three typical applications:group activity recognition,sensing data cross labeling and cross learning,and personalised image search.Specifically,the main research content and contributions of this dissertation are as follows.·Group activity recognition based on entity matching of multi-source sensing data.To address the problem that multiple sources of sensing data have different perspectives and a single data source has limited perspectives,this dissertation designs a skeleton matching estimation algorithm based on a greedy strategy and reconstructs the 3D scene based on it.To address the problem that in group activity recognition applications the global perspective often has multiple group activities performed asynchronously and it is too expensive to obtain a large amount of annotated data,this dissertation designs a graph-based human pose and person representation of human pose and human-object interaction,and then a bottom-up approach for activity recognition based on a statute approach.The experimental results show that the proposed method is efficient for group activity recognition,with an accuracy of 91.2%on M&MD,a dataset containing 10 categories of group activity data.·Cross labeling and cross learning based on multimodal sensing data entities and semantic matching.To address the problem of finding matching relationships between multimodal sensing data streams representing the same entity,this dissertation designs a matching method based on two semantic alignment metrics.To address the problem of accurately slicing continuous data streams according to the start and end times of actions and matching multimodal semantic data segments,this dissertation designs a multimodal adaptive slicing algorithm based on historical data segments and the recognition capabilities of existing models.To address the problem of conflicting prediction results caused by differences in the sensing dimensions of different sensing modalities data and incomplete model capabilities in cross labeling and cross learning application,this dissertation improves the traditional theoretical D-S algorithm,proposes a fusion algorithm that works well in the presence of conflicting predictions and completes cross labeling and cross learning of multimodal sensing data.The proposed framework achieves an average cross labelling accuracy of 98.5%on a multimodal dataset containing five different perception mechanisms:camera,smartwatch,smartphone,microphone and wireless access point.For the machine learning models of each modality,the recognition accuracy was 95.1%,91.8%,71.1%,91.2%and 34.4%,respectively.·Personalised image search based on multi-semantic text and image matching.To address the problem of complex matching relationships between multisemantic text and images,this dissertation proposes a knowledge graph construction method to relate the two modalities.To address the problem of the difficulty of users searching for images on mobile devices,this dissertation designs a semantic search mechanism and a hierarchical browsing mechanism based on the knowledge graph,and proposes a method to dynamically update the knowledge graph based on user interaction-aware data to implicitly learn user thinking patterns for generating personalised image search results.In the system evaluation,the search function saves users about 96%of time and 98%of operations,while the browsing function reduces 89%of browsing time and 93%of operations.