Research On Immersion Enhancement Method Of Visual-tactile Augmented Reality For Hand Function Rehabilitatio

Augmented reality technology is an innovative approach that overlays virtual information onto the real world,providing users with an enhanced perceptual experience.With the development of AR technology,integrating haptic perception into AR has become a focus of research,enabling not only the fusion of real-world scenes and virtual objects,but also the synchronization of visual and haptic perception.As next-generation information technology continues to advance,the potential of visuo-haptic augmented reality in hand function rehabilitation is gradually being revealed,with its engaging,interactive,and convenient features increasingly favored by patients with impaired hand function.During the training process,patients can both see and touch virtual objects situated in the real world,allowing their limbs to move in concert with the virtual items while receiving haptic feedback from them.This technology alleviates the tedium experienced by patients during treatment,effectively stimulating their haptic cognition,thereby achieving the goals of upper limb rehabilitation training and enhancing the effectiveness of rehabilitation outcomes.Supported by the National Natural Science Foundation of China,this paper investigates image restoration algorithms,depth map hole-filling algorithms,and haptic device calibration algorithms,and develops an immersive visuo-haptic augmented reality system for hand function rehabilitation.The main research contents of the article are as follows:Firstly,to avoid interference with user immersion caused by the presence of haptic devices,this paper proposes an image removal and restoration algorithm for use in visuo-haptic augmented reality environments.The algorithm consists of two stages: initially,image edges are detected using the Canny edge detection algorithm,followed by the prediction of edge images through an attention mechanism-enhanced edge generation network.The predicted edge images are then input into a texture restoration network to obtain the restored images.This model is deployed within the augmented reality program to enable real-time restoration of haptic devices..Secondly,the paper proposes an occlusion algorithm for eliminating depth map holes in both virtual and real scenarios.This method first identifies hole regions by analyzing pixel values.Subsequently,pixels in the corresponding RGB image area of the hole region are classified,and the holes are filled using similar pixels from adjacent areas,thereby achieving depth map enhancement.By calibrating the direction,position,and tip posture of the haptic device’s stylus,the virtual tool is accurately registered on the stylus,enabling the coordinated motion of the stylus and the virtual interactive tool.Finally,this paper applies visuo-haptic augmented reality technology to the field of rehabilitation.Based on the haptic device used,and in conjunction with augmented reality technology,a hand function rehabilitation training system with force feedback is designed.The system integrates the edge texture image restoration algorithm and the virtual-real occlusion algorithm,designing two different types of rehabilitation training programs:ring-fitting training and object-moving training.The ring-fitting training program aims to improve patients’ hand and wrist joint mobility and hand strength training,while object-moving training focuses on exercising patients’ finger grasping ability and haptic perception of force.Experiments have verified the feasibility and practicality of the designed methods in the ring-fitting training and object-moving training programs.