Study On Gesture Recognition And National Stringed Instruments Virtual Playing

The computer vision based gesture recognition has been widely applied in the field of human-computer interaction. It can take advantage of some correlation algorithm to analyze the image sequence captured by a camera and recognize gestures inside. Through the study of gesture recognition, this parer realizes a musical virtual performance system. It increases the musical fun, not only can attract more people to learn playing national musical instruments, but also contribute to the protection and inheritance of national intangible cultural heritage.This paper firstly introduces the related technologies used in the virtual performance system, including augmented reality, somatosensory interaction and gesture recognition, and compares several common computer gesture recognition technologies, then select the suitable vision based gesture recognition as our solution.Then the algorithm is designed and implemented based on the general process of gesture recognition. According to the actual situation of Erhu performance, on the left hand gesture recognition, this paper use the 3d model based gesture modeling method to model the five defined gestures, and use virtual gesture fitting method based on inverse kinematic to simulate and identify the basic gestures, it mainly introduces the best fitting location discovery algorithm; on the right hand gesture recognition, this paper use the appearance based gesture modeling method and Hu moment feature to model the two defined gestures, and use contour matching based recognition method to recognize the gestures.Finally, this paper realizes a gesture driven virtual Erhu performance system based on augmented reality. The system use Kinect as its input device, and makes full use of the color image, depth image, skeletal data captured by Kinect to build augmented reality scene. In the building process of the scene, through the depth data and a bayesian model in skin, the program get the user’s hand region, then draw the hand image on the enhanced image, thus solving the problem of occlusion between virtual and real worlds in augmented reality system. Finally, the paper shows the realization of the system.