Research On Somatosensory Interaction Based On Fusion Of Depth Camera And Accelerometer

With the continuous development of intelligence,human-computer interaction has undergone a revolutionary change.As a research hotspot in the field of human-computer interaction,somatosensory interaction conforms to the daily communication habit of human beings,and has wide application prospect in virtual reality,intelligent home,etc.At present,most somatosensory interaction systems perform human action recognition based on either an optical sensor or an inertial sensor,which may exist some limitations.Vision-based human action recognition need to address challenges such as occlusion,camera position,background clutter,and etcs.Therefore,the robustness of designed algorithms can not be guaranteed.In addition,the automatic segmentation of long-time continuous action is still a problem for vision-based method.Inertial-based human action recognition system need to place multiple sensor nodes on the human body,which may cause intrusion to the performer.In addition,the slip of the sensor nodes and the cumulative error during the movement can affect the recognition accuracy of the system.This paper presents a new human action recognition method based on feature-level fusion of depth camera and acceleration sensor.First of all,we built a data acquisition system using a depth camera and a group of accelerometers to realize the synchronous acquisition of color video,depth images and three-axis acceleration signal.After that,we released the Chinese Academy of Sciences—Yunnan University—multimodal human action database(CAS-YNU-MHAD).We implemented a spatiotemporal segmentation on the depth sequence and computed the time motion response of the obtained depth cube units,and then Histograms of Oriented Gradients(HOG)features were extracted as the depth image feature.We segmented the acceleration signal by sliding window and computed Fast Fourier Transform(FFT)coefficients of each window,and then all the FFT coefficients were concatenated as the acceleration signal feature.We maximized the correlation of corresponding features across two feature vectors and in addition decorrelated features that belong to different classes within each feature vector according to Discriminant Correlation Analysis(DCA).The uncorrelated fused discriminant feature was extracted by the obtained optimal projection transform as the final feature descriptor.Finally,linear SVM was used for model training and action classification.Experiments on CAS-YNU-MHAD show that the proposed method has a good performance in recognition accuracy and processing speed and is suitable for practical application of somatosensory interaction.