Research On Dynamic Gesture Recognition Technology And Its Application In Digital Twin System Interaction

With the continuous development of digital twin technology,the human-computer interaction under digital twin systems is receiving increasing attention from researchers.Immersion and interactivity are considered the most important characteristics in the context of the combination of virtual and real in the interaction of digital twin systems.Gesture interaction is more immersive,natural,and intuitive compared to traditional interaction methods.As a challenging problem,dynamic gesture recognition has many difficulties,such as complex gestures,complex environment and background,low accuracy and poor real-time performance.With the continuous development of deep learning and digital twin fields in recent years,dynamic gesture recognition shows a new application background and brings new technical solutions.On this basis,this article mainly conducts the following research:Firstly,conduct research on hand recognition and dynamic hand tracking issues,investigate current mainstream target recognition algorithms,and conduct comparative analysis of experimental results through experiments.In response to issues such as missed detections,false detections,and discontinuous tracking during the experimental process,a Kalman filtering algorithm was used for improvement.Based on this,a multi method fusion hand recognition and tracking algorithm based on YOLOv5 and Kalman filtering algorithm was ultimately implemented.Algorithm experiments were conducted on the constructed digital twin system to verify the algorithm.Secondly,the lightweight dynamic gesture recognition method based on attention mechanism and ConvNeXt module is completed.In order to solve the problems of low accuracy,poor real-time performance and many model parameters commonly existing in current gesture recognition algorithms,a lightweight design is carried out on the basis of ConvNeXt network.The lightweight experience of excellent lightweight models such as shuffle series and mobileNet series is absorbed to reduce the size of convolution cores.The depth separable convolution,grouping convolution and channel shuffling strategies are used to replace the general 3D convolution module A lightweight ConvNeXt module was designed to remove the reverse bottleneck layer,and a motion attention mechanism module was designed to enhance the motion feature extraction ability of the neural network based on the problem of poor temporal recognition in the experiment.The effectiveness of the designed modules was demonstrated through ablation experiments and verified on the constructed digital twin system.Finally,the digital twin system was built and the gesture recognition function was realized.According to the relevant theory of the digital twin five dimensional model,the digital twin system architecture of the manipulator was built.The system architecture was divided into five parts:physical entity layer,virtual model layer,data layer,connection layer,and functional service layer.The raspberry pie is used as the controller of the physical entity layer manipulator;Using Unity3D to model the virtual layer of the robotic arm,and using Unity 3D modeling to build the environment of the factory building to improve immersive medical examination;Using C#language for programming script control;Persistence storage of data generated by digital twins through MySQL database;TCP protocol is used to realize the secure transmission of data between the physical layer and the virtual layer,and finally gesture recognition control function is realized in the service layer.