Research On Visual Hand Pose Estimation Method Based On Deep Learning

In the remote operation control of robots,human-machine interaction based on devices such as handles,rockers,or control boxes is usually used,which in many cases appears to be not intuitive,natural,and flexible enough.At the same time,carrying control devices in some special scenarios can also bring certain constraints to the operator’s body.Due to the flexibility of human hands,using visual gesture-based interaction can overcome the above problems.Accurate hand pose estimation is a key link in achieving visual gesture-based human-computer interaction and is a research hotspot in the field of computer vision.To achieve human-machine interaction based on visual gestures,this thesis studies a three-axis pose estimation method for visual gestures based on deep learning,which mainly includes the following content.Firstly,to solve the problem of difficulty in estimating 3D hand posture from 2D visual images and weaken the interference caused by hand self-occlusion,this thesis proposes a fast construction method for a 3D hand posture dataset based on dual-view visual sensors and posture sensors.A fitting method based on the deep residual network was proposed for hand gesture 2D visual images to hand gesture three-axis pose data,achieving direct estimation of hand gesture three-axis pose angles from RGB images.Secondly,to improve the fitting accuracy,this thesis improves the performance of the proposed deep learning-based hand pose estimation model from both the algorithm and dataset levels.Using the idea of ensemble learning,two different ensemble strategies are proposed for the single convolutional neural network hand pose estimation model.This method can effectively improve the generalization performance of the model and improve prediction accuracy.Similarly,a dual view hand image pose annotation method based on Kalman filter virtual and real sensor data fusion was designed.The Kalman filter fused the predicted data output by the simulation hand pose estimation model with the observation data of the actual attitude sensor at the corresponding time of the real hand image,achieving high-precision pose data annotation of the hand pose dataset.Finally,based on the proposed hand pose estimation method,a remote operation control system for a two-degree-of-freedom pan-tilt system was designed,achieving intuitive control of hand attitude on the pan-tilt system.To improve the real-time performance of hand pose estimation,a YOLOv3-based hand detection model is designed and trained to detect human hands before hand pose estimation.Designed a hand pose estimation UI interface program based on Py Qt5 and Open GL to visually display hand pose estimation.In summary,this thesis addresses the issue of nonintuitive human-machine interaction in robot teleoperation control and designs a visual hand pose estimation method based on deep learning and a teleoperation control system based on visual hand pose estimation.