Research And Application Of Body Tracking And Interaction In Virtual Reality System For Skill Learning

With the development of computer vision and hardware technology,Virtual Reality(VR)applications with multi-dimensional perception based on the combination of multiple sensors are gradually replacing the traditional 2D desktop application and becoming the mainstream of information-based education.In the VR educational application for skill learning,it is necessary to provide an authentication service for learners so as to make personalized teaching programs for different learners.However,it is extremely inconvenient to use traditional Cipher-based authentication in VR environment.In the procedure of skill learning,teachers and students need to have a lot of interactions,such as raising their hands to ask questions,monitoring the operation of students,etc.But the operation instructions of the traditional VR system are relatively simple,which can only change the spatial position and rotation of the operating object,and can not provide rich feedback for the participants' actions.As the only controller for traditional VR applications,the handle has a low degree of freedom to customize its functions,which limits the fidelity and scalability of VR collaborative learning applications.This thesis presents a face recognition method based on sparse constraints for authentication tasks in VR applications.To address the problem that the degree of freedom of function customization can not meet the needs of operation diversity in VR collaborative learning scenes,a scene control method based on human action recognition is proposed.The main contributions of this thesis are as follows:1.A face recognition algorithm based on sparse constrained convex nonnegative matrix factorization is proposed.The algorithm extends the dimension of matrix factorization,maintains the manifold structure and improves the anti-noise ability of the algorithm.2.RGB feature,Depth feature and Joints feature are combined as a high dimensional fused feature to describe the posture of human body to complete the action recognition.The data from a variety of sensors are fully utilized and the three features are effciently fused based on Late-fusion.3.An orthogonal constrained unsupervised feature selection algorithm with hypergraph regularization is proposed.The algorithm retains the hypergraph-based manifold structure and learns more spatial structure information of samples while reducing the combined feature's dimension.4.A small action recognition dataset is built.The dataset contains samples of six common actions,with large differences in motion's magnitude and body's height.Through the depth camera,RGB images,depth images and keypoints of human body are collected.Manual labeling of each sample in the dataset provides the ground-truth of the action category.In the aspect of action recognition,cluster experiments were performed on public facial datasets called Grimace and Faces95.8 face recognition algorithms based on nonnegative matrix factorization are analyzed and compared.The proposed face recognition algorithms are superior to the comparison algorithms in varying degrees.In the aspect of action recognition,the action recognition algorithm proposed in this thesis achieves an average correct recognition rate of 88% on the data set,which can basically meet the scene control requirements of VR applications after being mapped to control instructions.