Research On Elderly Behavior Recognition Method Based On Multimodal Graph Convolutional Neural Networ

With the problem of the aging population becomes serious,we pay more attention to the safety of the elderly when they are at home alone.In order to provide early warning,alarm,and report of some dangerous behaviors,several domestic and foreign research institutions are focusing on studying the intelligent monitoring of the daily activities of the elderly in robot-view.For promoting the industrialization of these technologies,this work mainly studies how to automatically recognize the daily activities of the elderly,such as "drinking water","washing hands","reading a book","reading a newspaper",etc.Through the investigation of the daily activities videos of the elderly,we found that the semantics of the daily activities of the elderly are obviously fine-grained.For example,the semantics of "drinking water" and "taking medicine" are highly similar,and only a small number of video frames can accurately reflect their category semantics.In order to effectively address the problem of the elderly activity recognition,we propose a new Multimodal Multi-granularity Graph Convolutional Network and a Multi-stage Fusion Full-modal Graph Convolutional Network by applying the graph convolution networks on these modalities.The work of this paper is mainly summarized in the following three aspects:(1)A new Multimodal Multi-granularity Graph Convolutional Network is proposed.The method is mainly composed of two modules,which are the Multi-granularity Graph Convolution Networks(S-GTCNs)module based on the skeletal modality and the Multi-granularity Graph Convolution Networks(R-GCNs)module based on the RGB modality.S-GTCNs obtains the representation of activity feature by using the bone data and skeleton data,and obtains the output of S-GTCNs by applying the graph convolutional network including the attention mechanism.R-GCNs extracts the segments of the video through the Boundary Sensitive Network(BSN),and samples the key frames,and then obtains the corresponding output through the attention-based graph convolution network.The experimental results show that the elderly daily activity recognition method based on the Multimodal Multi-granular Graph Convolutional Network proposed in this paper can effectively improve the accuracy of activity recognition.(2)A "point-line-frame-proposal" four-granularity data representation strategy is proposed,and the information complementarity of multi-granularity data is used to finely describe the fine-grained elderly behavior in the video."Point" represents the bone sequence,"line" represents the bone sequence,that is,the sequence obtained by subtracting the bone sequence,and "frame" represents the key frame in the RGB video,"proposal" represents the segment in the RGB video.By fusing the information between the two modalities,mining the complementarity and consistency among them to improve the learning ability of the model.By dividing a variety of fine-grained data and fusing the focused information contained in different granular data for learning and training,the model can be comprehensively guided.(3)A Multi-stage Fusion Full-modal Graph convolutional Network is proposed,which adds the processing and application of Depth Map modal.Because the skeleton data only focuses on the action itself,it ignores the interactive information between people and objects,and cannot distinguish objects.Although RGB video can identify the texture and shape of objects,RGB video is greatly affected by lighting,background,and occlusion.Depth Map can easily distinguish foreground and background,not affected by lighting,and is also image data.Therefore,RGB The modal data and the Depth Map modal data are early fused to obtain the recognition scores based on the two modalities,and then the late fusion with the recognition scores based on the skeletal modalities is performed to obtain the final recognition results of the elderly’s daily behaviors.Experiment results show that this method can achieve significant performance improvement.