| As an important basic task of a service robot arm,liquid pouring not only has a broad application prospect in service situations,but also has humanitarian significance for vulnerable groups such as the elderly and the disabled,with important research and application value.The liquid pouring process of the robot arm is similar to that of humans,which can be divided into three links: grasp,transporting,and pouring.Based on the theory and technology of computer vision,this thesis focuses on the issues of grasp detection for liquid bottle,deduction on position for pouring,liquid level detection and control,etc.The main research contents are as follows:Firstly,based on the analysis of the scenes in which robot pours liquid,a visionbased liquid pouring experiment platform was established with the Jaco2 robotic arm and the Real Sense D435 i depth camera as the hardware,ROS as the software.The forward and inverse kinematics of the Jaco2 robotic arm,the camera imaging model,and two hand eye calibration methods,Tsai-Lenz and EPNP,were analyzed and derived to achieve both eye-in-hand and eye-in-base calibration.The overall work plan is constructed with liquid bottle grasp link based on reconstruction of point cloud from instance segmentation and depth map for computing grasp pose,transporting link based on the deduction of the liquid pouring position through object detection,and the liquid level detection& control link based on object detection.Secondly,with the issues on the balance between speed and accuracy,generalization of the detection for liquid bottles placed in various service environments in the grasp link considered,a method for reconstructing point clouds of liquid bottle based on instance segmentation and aligned depth maps is proposed.Mask R-CNN segmentation masks are used to determine the area for reconstruction,reconstruct points within the area,and skip points outside the one.Based on the proposed,combined with analysis of the geometric commonality of liquid bottles and the structure of the gripper,the grasp pose for liquid bottles is designed,and a method for estimating the grasp pose of the liquid bottle considering the centroid distance and the constraint of the normal vector is proposed.The grasp points are scored by the combination of the cosine of the angle between the normal vector and the z axis of grasp coordinate with the centroid distance to obtain a reasonable grasp pose.The segmentation and reconstruction experiments have verified that the instance-segmentation-based method on point cloud reconstruction for liquid bottle has certain generalization for different environmental conditions such as lighting,table background,can accurately reconstruct liquid bottle point clouds,and saves huge amounts of calculations on unrelated points.The grasp experiment has achieved effective grasp of liquid bottles placed in multiple poses.The success rate of grasp for liquid bottles placed in three poses,namely reclining,lying down,and upright,is higher than 75%.Then,aiming at the problem that noise interference such as lighting,color or background of the table and so on,affects the detection effect of liquid bottles,cups,and liquids in robot service situations,combining with the requirement that the deployment is to be lightweight,an object detection model YOLO-Uni based on CA attention,lightweight Ghostconv module and C3 Ghost module for feature extraction is proposed.The model synthesizes important features of channels and spaces,accelerates the convergence using EIOU loss functions.Through collecting relevant images from COCO and VOC datasets,downloading relevant images from the network,and collecting images on the experiment platform,a common robot liquid pouring visual dataset shared to both instance segmentation and object detection is obtained,including three types of objects: liquid bottles,cups,and liquids.The experiment shows that the detection accuracy of YOLO-Uni in the liquid pouring vision dataset in experiment environment is 97.4%,which is 7.4% and 5.1% higher than YOLOv5 s and Fast R-CNN,respectively.The parameter amount and FLOPs are 64.2% and 64.7% of YOLOv5 s,respectively.During training,the positioning loss of YOLO-Uni decreases faster than YOLOv5 s.Aiming at the requirements for the convergence and rapid detection of liquid bottle transportation and liquid pouring links,a unified detection method based on object detection for transportation position and liquid level is proposed.Based on YOLO-Uni,a reasonable transportation position for liquid pouring is obtained through the information of the detected object combined with coordinate conversion for the deduction of the center,height and width of the cup and the height of the bottle.A planar-geometric-analysis-based algorithm is designed to determine the spatial inclusion relationship between the cup and the liquid in the cup,the ratio of the pixel heights of the liquid to the cup is availed as the liquid level for detection,avoiding the complicated calibration required for absolute value measurement.The experiment of liquid pouring position deduction shows that the success rate of the unified detection method can reach 90%.Build an experiment platform based on Baxter dual-arm robot to verify the generalization among scenes of liquid level detection& control methods.The experiment results show that the unified detection method can support liquid level detection under various camera heights,inclinations,and placements.The accuracy of the model for detecting multiple cups and liquids in the experiment is higher than 80%under different lighting,table color texture,and certain occlusion conditions.The model trained on the Jaco2 platform still has certain generalization when deployed to the Baxter platform for detection.Finally,aiming at the generalization and robustness of liquid level control in the liquid pouring link,a unified visual closed-loop control method for liquid level is proposed.The visual detection of liquid level is put into the feedback loop to form a closed-loop control with the control algorithm.The difference between the expected liquid level and the detected one is used as the input,and the angle of the end as the output through PD control to make the liquid level match the expected value.Liquid level control experiments were both carried out on the single-arm Jaco2 and the dualarm Baxter robot platform,with one set of PD parameters achieving liquid level control at multiple expected levels from 0.2 to 0.8(such as 0.25,0.4,0.5,0.6,0.75,0.8,etc.in liquid height/cup height)on the both.The error between the mean results of the experiment and the expected values was 0.01(liquid height/cup height)at the optimal situation,verifying the robustness and generalization of the proposed.Ultimately,realize the whole liquid pouring process connecting each link on the platform with success rate reaching 70% on the plain white table.The thesis contains 97 figures,26 tables,and 124 references. |
