| With the intelligent upgrading of the traditional industries,more and more intelligent equipment has been applied to the industrial field.For example,the mobile robots are used to cooperate with manipulators to achieve material sorting and distribution in the intelligent storage systems.In such unstructured environments,it is necessary to increase visual perception capability for the entire system to ensure safe operations.This paper proposes a deep learning method to estimate the 3D pose and predict the short-term motion of the mobile manipulator based on an RGB camera in the case of double manipulators cooperation.The specific contents of this paper are as follows:(1)Aiming at the 3D pose estimation for a manipulator,a method for rapid construction of dataset based on the real manipulator is proposed.This method uses automatic annotation,which can significantly reduce the cost of dataset annotation.Each data sample includes an RGB image and a depth image of the real manipulator,joint angle data,and pixel coordinates of the defined key points.The structure of the dataset is clear and standardized,which facilitates the subsequent expansion of the dataset.(2)Based on the manipulator dataset,a staged method for manipulator pose estimation is proposed,which can estimate the 3D positions of the key points on the manipulator based on an RGB image in real-time.This method is mainly divided into three stages: target detection,2D pose estimation,and3 D pose estimation,which can overcome the problems of end-to-end network such as lack of the 3D training data and the implicit modeling of the camera parameters.Our method first implements the target detection in the scene of the manipulators based on the YOLOv3 model,then improves the traditional pose estimation network to achieve accurate detection of the2 D key points of the manipulator,and finally uses the manipulator kinematics model and optimization algorithm to achieve the 3D pose estimation of the manipulator.Experiments show that the real-time performance and effectiveness of the method in each stage have been fully verified on the manipulator dataset.(3)Based on the 3D pose estimation of the manipulator,a deep learning method for predicting the short-term movement trend of the manipulator in the joint-Cartesian space is proposed.Combined with the cooperation scenario of the double manipulators,a collision avoidance strategy for the slave manipulator is proposed to achieve the safe cooperation of the double manipulators system in the unstructured environment.This method combines the kinematics model of the manipulator to transform the motion prediction task to the joint-Cartesian space,which reduces the complexity of the network and the difficulty of learning.Based on the pose estimation and motion prediction of the manipulator,the interpolation method is used to extend the obstacle space of the master manipulator,and a corresponding slave manipulator collision avoidance strategy is proposed.The experimental results show that the model can accurately predict the movement trend of the manipulator,and the simulation experiments also verify the rationality of the collision avoidance strategy.(4)In order to practically verify the effectiveness of the method for real-time 3D pose estimation and motion prediction of the mobile manipulator based on the visual information,as well as the effectiveness of the double manipulators safety cooperation method,a double manipulators cooperation platform including the mobile manipulator is set up.The corresponding software framework mainly includes the mobile manipulator control module,the system vision sensing module,and the slave manipulator motion planning module.The final experimental results show that our method for pose estimation and motion prediction of the mobile manipulator based on the visual information,combined with the collision avoidance strategy of the slave manipulator,can achieve the safety cooperative task of the double manipulators system. |
PDF Full Text Request