The Research On Activity Recognition Method Based On Multi-source Multi-modal Data And Multi-dimensional Convolutional Network

With the development of deep learning,human activity recognition(HAR)technology has received increasing attention,an increasing number of researchers have focused on studying deep learning methods.Moreover,the improvement of hardware performance of various wearable devices provides a theoretical and practical basis for the popularization of sensorbased HAR technology.At present,sensor-based HAR technology has been successfully applied to motion analysis,smart home,special group monitoring and other fields.In order to obtain higher recognition accuracy,most current HAR studies involve multisource and multi-modal sensors(MMS)data.Despite many achievements in this area,sensorbased HAR technology still has many shortcomings.In terms of datasets,the existing public family activity datasets usually contain only a few kinds of daily activities,which are difficult to simulate the smart home scenario;In terms of feature extraction,the existing algorithms generally use the convolution kernels of a single-dimension,due to the limitation of the receptive field of the convolution kernels,these networks are still defective in extracting the relationship between the MMS data;In terms of practical application,human activities are diverse and complex,which are easy to be confused.It is difficult for existing networks to make accurate judgments.In regard of the above problems,the key contributions of this thesis are as follows:(1)Aiming at the simple types of daily activities in the existing public dataset,a variety of daily activity data are collected through customized acquisition terminals to construct daily home activity(DHA)dataset,which can be used to smart home and other scenarios.(2)Aiming at the problem of single dimension of feature extraction in HAR network,a multi-dimensional parallel convolutional connected(MPCC)network structure is proposed.The structure uses the receptive fields of convolution kernel of different dimensions to extract multi-dimensional features.Enhanced correlation between multi-sensor data.In addition,multiscale residual convolutional squeeze-and-excitation(MRCSE)modules are proposed to enrich the diversity of feature information by combining squeeze-and-excitation(SE)blocks.The recalibration of the adaptive relationships of the feature maps is completed while avoiding network degradation.Finally,an MPCC network based on the MRCSE modules(MPCCMRCSE)is proposed.The network combines the advantages of the first two,and achieves FW-scores of 98.33%,97.07%and 95.92%on the physical activity monitoring(PAMAP2),OPPORTUNITY and DHA datasets using 10-fold cross-validations,respectively,which are better than the latest and classical network models.(3)Aiming at the wide variety and confusion of human activities in practical applications,this thesis combines recurrent neural network(RNN)with MPCC-MRCSE network,the Monitor network is proposed.The activity characteristics extracted from the pretrained MPCCMRCSE network are continued to be transmitted to the bidirectional gated recurrent unit(BiGRU).By acquiring the context features of the multi-sensor,the time relationships of the multi-sensor data are further extracted.The Monitor network achieves FW-scores of 99.01%,98.75%,and 96.39%on the PAMAP2,OPPORTUNITY and DHA datasets using 10-fold crossvalidations,respectively.Compared with the MPCC-MRCSE network,the performance is significantly improved.Through the above research,the effective recognition of complex human activities and confusing activities is realized.Through experimental analyses on the PAMAP2,OPPORTUNITY public dataset and DHA self-collected dataset,the effectiveness and superiority of the proposed network and method are proved.HAR network proposed in this thesis can be applied to many HAR scenarios such as smart home,which has outstanding research value and application prospect.