Design Of A Natural Human-hand Interaction Rehabilitation System Based On Augmented Realit

Augmented reality technology is widely used in education,industrial maintenance,medical,and other fields.The application of augmented reality human-computer interaction technology to hand function rehabilitation has gradually become a research focus.The purpose of this paper is to study the hand posture estimation algorithm based on deep learning,including the estimation of two-dimensional and three-dimensional hand posture,and a combination of them with augmented reality technology to design and develop a hand function rehabilitation training system based on augmented reality.The main research contents of this paper are as follows.Firstly,this paper proposes a YOLOv3-HM algorithm based on deep learning target detection and heatmap regression.Based on the commonly used target detection algorithm YOLOv3,the algorithm improves part of the network structure,combines heat map regression to achieve two-dimensional hand pose estimation,and returns the joint point coordinates of the hand.The algorithm is tested on the Frei Hand dataset and real-time video images.The results show that,compared with the traditional algorithm,the algorithm proposed in this paper improves both the attitude estimation accuracy and the tracking effect.Secondly,this paper proposes an end-to-end 3D hand pose estimation algorithm based on the hand parameterized model MANO.The images are sent through a convolutional encoder to an iterative 3D regression module,and the algorithm infers the parameters required by the MANO model.The MANO model will estimate the hand pose based on these parameters.Next,the 3D parameters are fed into the discriminator to judge whether these parameters come from real hand shapes and poses.After experimental verification,the algorithm in this paper has good accuracy and robustness for 3D pose estimation of the hand.Finally,this paper combines hand posture estimation technology with augmented reality registration technology.A hand function rehabilitation system based on augmented reality is designed and developed.At the same time,three rehabilitation training items with different levels of difficulty are designed,that is,jigsaw puzzle training,square game training,and barbell game training.The feasibility of the rehabilitation training plans is analyzed based on experimental results.