The Research On Human-Machine Cooperative Operation Of Dexterous Robot Hand Based On Vision

Robots as the most direct application of artificial intelligence,play an important role in many fields such as industry,agriculture,medical,and aerospace.And the ability of the robot to manipulate is an important manifestation of its intelligence.However,it is difficult to make autonomous decision-making in the face of complex scenes in an unstructured environment.Especially for robot hands with high degree of freedom,it is more difficult to plan independently.Therefore,they need to learn from human demonstrations.Traditional methods use data gloves or wearable sensors to obtain human hand posture parameters,where the device system is cumbersome and affects the natural movement of the human hand.In order to obtain an intuitive and natural human hand poses,this thesis adopts a vision-based non-contact method to obtain human hand movements through RGBD camera,based on deep learning method to predict human hand poses,and map them to dexterous robot hand,so that the dexterous robot hand can imitate the flexible hand movement of human,which enables the human-robot cooperative operation.The main work of this thesis is listed as follows:(1)Given consideration to both the accuracy and the real time performance,this thesis designs a novel three branch Convolutional Neural Networks named Hand Branch Ensemble network(HBE),where the three branches correspond to the three parts of a hand: the thumb,the index finger and the other fingers.The structural design inspiration of the HBE network comes from the understanding of the differences in the functional importance of different fingers.In addition,a feature ensemble layer along with a low-dimensional embedding layer ensures the overall hand shape constraints.The experimental results on three public datasets demonstrate that our approach achieves comparable or better performance to state-of-the-art methods with less training data,shorter training time and faster frame rate.(2)A complete human-robot cooperative control system is designed and implemented.The experiment is implemented on Simox robot simulation platform and Shadow Hand robot.The whole process is divided into four main modules: real-time image frame acquisition and preprocessing,hand pose estimation,human-robot hand poses mapping and human-robot cooperative manipulator control.According to the physical structure of the hand joint,the joint angle of the human hand is obtained by inverse kinematics solution.The joint angles of the human hand are corresponding to that of the dexterous robot hand one by one,so that it can be controlled to imitate the flexible actions of the human hand and realize the human-robot coordinated control based on vision.(3)As for the hand pose estimation from hand-object interaction,this thesis designs a novel hand pose estimation network based on the hand branch ensemble network.The parallel cross-resnet block extracts rich global depth features,and the multi-modal feature fusion module is used to combine the high-level semantic features of color images with the low-level spatial features of depth images,and the parallel interference elimination module of branch features is used to obtain more pure branch features.In the occlusion environment from hand-object interaction,this network can estimate the 3D positions of all the hand joints.