Research And Implementation Of Human-Computer Interaction System For VR Games Based On Computer Vision

Human-computer interaction has always been a popular research fields in industry and academia,and with the popularity of the meta-universe,technology of human-computer interaction has become more and more important.In gaming entertainment,VR games is a popular form of game.Traditional VR games rely on assistive devices such as controllers,expensive LIDAR and RGB-D cameras,which would increase the player’s burden.Considering these,a method of computer vision-based human-computer interaction is proposed.This method only requires two common RGB cameras to complete the interaction between the player and the VR game.Relying on the 3D cloud gaming platform,this thesis takes a third-person game as the object of study and accomplishes the following contents:(1)Research of 2D human key-points detection algorithm.In order to control the character in the game,the human key-points detection method is used.In order to make the human key-points detection algorithm satisfy the requirements of the application,the Shuffle Netv2 Block is used at the feature extraction stage in PIFPAF model,the loss function and some parameters are also optimized.After these improvement,the NE reaches0.211% and the OKS reaches 0.976,the algorithm becomes more lightweight and can train and run faster than before.When running on RTX2060 GPU,the speed of detection can reach 13 fps if the size of image is 320*240.(2)Research of dimension-raising algorithm.In order to solve the problem of traditional human key-points detection algorithm that can only obtain the 2D coordinates of the human key-points in the pixel coordinate system,a dimension-raising algorithm based on binocular vision is proposed.The method firstly detects the 2D position of human key-points in the images captured simultaneously by the two RGB cameras,then combined with the internal and external parameter of the camera,through geometric calculation,the 2D position can be transformed to the 3D position.After quantitative testing,under the experimental conditions in this thesis,the accuracy of this dimension-raising algorithm can reach the centimeter level,and gets good visualization results.(3)Research of gesture recognition algorithm.An improved GBDT static gesture classification algorithm is designed to achieve the recognition of 11 static gestures for the purpose of player’s interaction in the game through hand movements.By setting different weights for difficult and easy classification samples in training,better test results can be obtained.The recognition of game actions such as "pick up" and "shoot" is achieved through analysis the relationship of static gestures changing between video frames.(4)Design and implementation of the game interaction system.The 3D human body key-points obtained through the algorithm is encapsulated in JSON format and uploaded to the cloud game running platform at each frame via network communication,and the rendering results from the cloud game running platform are received for playback.The control of the character in the game by the game player is realized,the purpose of human-computer interaction is reached and gets a good effect.