Algorithm Of Computer Vision-based Hand Gesture Recognition And Application Of Human-computer Interaction

Hand gesture interaction is an important and hot research topic in human-computer interaction because of its naturalness and intuition.Technologies of hand gesture interaction can be classified according to the input devices,such as data glove,acceleration sensor,touch screen,monocular camera,depth camera and so on.This dissertation is dedicated to the hand gesture interaction technology based on monocular vision which identifies hand gestures by analyzing the images of bare hands.This technology conforms to human habits because no extra device is required.However,the limited input information leads to some challenging problems.For example,the accuracy and efficiency of hand gesture recognition is sensitive to the high degree of freedom of human hand and the complexity of the scenario.In this dissertation,several algorithms of hand gesture segmentation and recognition are proposed to solve the problems mentioned above,especially the influence of human faces.Furthermore,a real-time finger mouse system which achieves some functions of the mouse is developed by using the proposed algorithms.The main content and contributions of this dissertation is summarized as follows:(1)For hand recognition under the complex scenario,hand gestures are segmented by combining skin color extraction and background subtraction methods.We build a histogram-based model of skin color by using the chrominance information in YCbCr color space and a background subtraction model combining temporal differencing.Since the colors of the hands are various at different locations and the presence of other skin-like objects and dynamic scenarios make the discrimination of the bare hand difficult,both the skin color model and the background model are updated in real time.Then a new support vector machine algorithm based on Hu moments,convexity and compactness of hand contours is proposed to recognize six hand postures.Experimental results show that the target regions can be extracted fast and accurately in the complex background.Furthermore,the proposed method achieves a high recognition rate of 98% for the discrimination of hand postures performed by different users and the geometric transformation of hands.(2)Since the user's face appears in the camera view frequently,the efficiency of hand recognition is largely influenced by the face because of the uncertain movement and the similarity of color and texture to the hand.Therefore,how to distinguish the hand from the face is one of the most important parts in this dissertation.For the influence of human face,an edge repair-based hand subpart segmentation algorithm is proposed to segment the hands.First,a hierarchical chamfer matching algorithm is employed to locate the hand region roughly.Second,in consideration of the flexibility of the fingers,the hand is divided into palm and fingers,which are separately detected by combining pictorial structures and HOG characteristics.Since edge information is probably blurry at the border of the hand and face,a completely connected curve representing the hand silhouette is difficult to figure out.Consequently,an edge repair method based on pre-stored templates is presented for each subpart.Finally,the repaired contours are combined to extract the precise hand region.The experimental results show that our algorithm is robust to the movement of the head,different users and geometric transformation of hands.TPR and FPR are 94.6% and 8.7% in our hand dataset.(3)In order to improve the efficiency of hand segmentation,deep learning is introduced to our hand gesture interaction system and a hand segmentation method based on convolutional neural networks is proposed.Deep learning is a breakthrough technology in machine learning and artificial intelligence.Features can be learned autonomously from original images instead of manual extraction.After multiple experimental measurements,a network structure based on fully convolutional networks is designed which mainly contains four convolution layers and one deconvolution layer.Experimental results demonstrate that the proposed method solves the interference problems caused by other parts of human body,different illumination and complex scenarios.The method is robust to individual difference and geometric deformation of hand gestures.TPR and FPR reach 93.8% and 4.3% which are improved relative to the conventional methods.Besides,the operation speed of our deep learning model is 18 ms on GPU,which satisfies the real-time requirement.(4)For three platforms of PC,mobile phone and smart TV,a finger mouse system based on monocular vision is developed by using the proposed algorithms of hand segmentation and recognition.We design the interaction mode,the system framework and the algorithm flow and propose an adaptive cursor positioning method based on the gazing direction.Users can communicate with different platforms with predefined hand postures to realize the mouse functions,such as moving the cursor,moving a document,zoom in/out,click,copy and paste.Due to the adaptive cursor positioning method based on the gazing direction,the cursor can be moved fast and accurately by estimating the relative positions of human eyes and fingertips.Moreover,the positioning error caused by lacking of depth information is adaptively corrected.Experimental results show that the system works successfully and offers a natural and comfortable interaction experience.