Research And Application Of Gesture Recognition In The Immersive Panoramic Stereo Video Living System

Virtual Reality(VR)technology has been successfully applied in many fields,such as education,tourism,games,etc.Some researchers hope to combine VR to watch panoramic stereo video and bring a brand new living experience in the field of multimedia.However,in practical applications,there are many problems such as panoramic video real-time splicing and high-resolution video compression coding,so it is still difficult to construct a panoramic stereo living video system.In this thesis,we design and implement a panoramic stereo video living system.Based on the eight-eye panoramic annular lens,the 360°binocular video is stitched.Users wear VR helmets,watch stereo panoramic video living and feel the brand new experience.At the same time,because of the inconvenience of hand grip interaction in VR,we further studied the vision-based gesture recognition technology,combined with Leap Motion gesture somatosensory acquisition camera,realized the recognition of predefined gestures in the panoramic video living system and completed the interaction of video scenes in the system.The work of this thesis mainly includes:1.Combined with convolution neural network,multi-modality fusion temporal segment networks(MMFTSN)is proposed for gesture recognition.In this model,three gesture modalities: RGB,Depth and optical flow video data are equally divided,randomly sampled.Then,the extracted frames are classified using a convolutional neural network.Finally,fusing three kinds of modality classification results.Our model used to obtain the recognition rate of 60.2% on the gesture dataset Chalearn LAP Iso GD,which was better than the performance of related algorithms.The results show that the improved performance of the proposed model.2.Design and implement a panoramic stereo video living system.The binocular panoramic video is acquired and spliced in real time by using multiple cameras.The H.265 encoding algorithm was used to compress and encode the video more efficiently.The video stream is pushed to the cloud server for forwarding.The client uses VR helmet and combines with Unity development tool to receive and watch panoramic stereo video in real time.3.Using Leap Motion depth camera,combining VR helmet and Unity,a virtual gesture interaction experiment is designed and completed at the receiving end of the living system.Position information of gesture key points is obtained through the Leap Motion gesture sensor,then,the distance and speed features are defined based on the position information of these key points,corresponding gestures are identified through these fatures,so as to realize video rotation,scaling and movement operations in a living system are completed.In this thesis,gesture recognition technology is applied to panoramic stereo video living system.The experimental results show that the system can operate normally and recognize predefined gestures.Gesture interaction in the living system is completed and a more natural human computer interaction mode is realized.It provides a solution to replace hand grip interaction in VR.