Hand Action Recognition Based On NST-GCN With Deep Learning

Machinery manufacturing continuously toward the direction of more intelligent,more and more people pursue personalized,increases demand for personalized custom small batch manufacturing,production mode with the corresponding changes have taken place,has brought the change of mechanical manufacturing,with the mature of human-computer interaction technology,provides the intelligent machinery manufacturing flexibility and efficiency of effective measures.Hand motion recognition is based on the intuitive and informative hand posture recognized by computer as input,and the direct interaction between robot and human is realized through decision algorithm and cooperation strategy.In this paper,the constructed network models of hand object detection,hand pose estimation and hand action recognition are used to study hand video and complete the research of hand action recognition.Finally,the performance of the network model is evaluated by experiments in real scenes.A hand target detection model based on NYOLOv4 was established to determine the specific position of the hand in the image.On the basis of YOLOv4 target detection network,a hand detection network was constructed and a prior frame for hand detection was preset by k-means clustering method.The backbone network was improved to reduce the number of parameters of the network,and then the multi-scale feature fusion structure was used to improve the performance of the network,and a hand detection model based on NYOLOv4 was established.The hand detection model based on NYOLOv4 has little difference in accuracy compared with the original model,but it runs more than twice as fast as the original model.A hand pose estimation model based on NNSRM was established to estimate the coordinates of the key points of the hand based on the target position of the hand.Firstly,based on HRNet network,an attention channel is added by integrating SA(attention mechanism)module to make the network pay more attention to the key points of the hand,so as to build the NHRNet model.Then,based on the NSRM network,the backbone network is replaced by NHRNet,and the input and output channels of part of the convolutional layer are reduced to achieve more accurate and faster hand pose estimation,so as to build the NNSRM model.Experimental results show that compared with the original model,the NNSRM hand pose estimation model has improved accuracy and reduced the number of parameters,which can accurately estimate the coordinates of the key points of the hand.The dynamic gesture recognition model based on NST-GCN is established,and the key points of hand are modeled to realize the dynamic gesture recognition.Firstly,the spatiotemporal convolution is constructed and the sampling function and weight function are set up.Then,three modules are proposed,including dilated convolution,non-physical connections and a new partition strategy.The effectiveness of the three improved modules is verified by ablation experiments,and the NST-GCN network model is constructed.The experimental results show that the NST-GCN dynamic gesture recognition model has better accuracy and recognition effect than the original model.The experiment of hand movement recognition in mechanical environment was carried out.The hand target detection model hand pose estimation model and hand action recognition model are combined in series,and mobile phone cameras were used to collect 14 types of action videos.For each gesture,2 testers collected 10 action sequences for experimental testing,and then the hand movements in the production videos of two factories were identified.The test results show that the hand target detection model in this paper can detect the specific position of the hand part well in the mechanical environment.In this paper,the hand pose estimation model can identify the key points of the hand part well without occlusion,and predict the key points of the hand part with good robustness in the case of occlusion.The hand motion recognition model in this paper needs to be improved in recognition of gestures with similar trends.The model established in this paper can realize hand motion recognition in actual production and manufacturing environment,and the relevant results obtained in this paper can be applied to man-machine collaborative axle hole and gearbox assembly.By recognizing the operator’s hand movements,the robot can cooperate with man-machine collaborative workpiece loading and unloading and assembly,greatly improving the flexibility and efficiency of complex operations.Other applications include service robots,emergency rooms and operating rooms.