Research On Three-dimensional Visual Perception And Obstacle Avoidance Methods Of Harvesting Robots

Fruit harvesting is a labor-intensive job and is greatly affected by labor shortages.Under the background of the declining agricultural age-appropriate labor force in China,it is urgent to develop intelligent fruit harvesting robots in order to ensure timely fruit harvesting.There are a large number of smallholder operating orchards in China.The management methods of these orchards are relatively backward,resulting in widespread branches and leafy leaves,which may increase the difficulty of robot picking.Therefore,robust three-dimensional(3D)fruits and obstacles(such as branches,trunks,and poles)detection is a key technical problem in the current research and development of harvesting robots.In addition,in order to avoid collisions between robots and obstacles and thus improve the harvest success rate,it is necessary to study how to use 3D fruit and obstacle information to effectively guide the robot to avoid obstacles.Aiming at solving these two problems,this paper takes computer vision,artificial intelligence,optimization theory,and robotics as the basic theoretical methods to deeply study 3D fruit detection,3D obstacle detection,and obstacle avoidance path planning.The research contents and conclusions are given below.(1)Aiming at the demand of robust 3D fruit detection for harvesting robots,a depth-based 3D fruit detection method and a depth-and color-based 3D fruit detection method were proposed.The former first uses region growing to cluster the depth image into superpixel blocks,then applies a sphere detection algorithm to detect candidate fruits from these blocks,and finally deploys a support vector machine classifier(SVM)to identify false positives.The latter first runs a Bayesian classifier to coarsely segment the red-green-blue-depth(RGB-D)image,and then uses density clustering to group the RGB-D image into non-subdividable point clouds,and finnaly deploys a SVM to exclude false positives.Experiments showed that the F1 scores of the first method on the guava and citrus datasets were 0.839 and 0.833,respectively;the second method were 0.921 and 0.919,respectively;their positioning errors in the x,y and z directions were 6.5±2.5mm,-3.0±3.0mm and 12.0±4.0mm,respectively.In conclusion,the developed 3D fruit detection methods are robust and accurate,and can meet the requirements of fruit harvesting robots;the second method is more robust than the first method.(2)Aiming at the demand of robust 3D obstacle detection for harvesting robots,a new idea was proposed to approach irregular obstacles with 3D line segments.On this basis,two3 D line segments detection methods were developed,one based on the full convolutional neural network(FCN)and random sampling consistency algorithm(RANSAC)and another based on Mask R-CNN.Method one first employs FCN to segment the RGB image,then skeletonizes the segmentation result and converts it into a point cloud,and finally uses RANSAC to extract 3D line segments from the point cloud.Method two adds a 3D line segment regression branch on Mask R-CNN.Through end-to-end training,the network can realize obstacle detection,segmentation and localization simultaneously.Experiments showed that under the relative distance error and relative reprojection error measurements,the F1 scores of method one were 0.4608 and 0.4822,respectively;method two were0.4005 and 0.5778,respectively.In conclusion,the two methods have good robustness and can meet the detection needs of picking robots for obstacles;the comprehensive performance of method one is better than method two.(3)Two obstacle avoidance path planning algorithms were investigated in order to solve the problem of obstacle avoidance picking.The first algorithm is based on improved artificial potential field(i APF).It constructs a Cartesian space force based on the 3D information of fruits and obstacles.Robot jacobian is then used to transform the Cartesian space force to a joint space force.Afterwards,a gradient descent algorithm is performed to change the joint space force to promote the robot to avoid obstacles.The second algorithm is based on recurrent deep deterministic policy gradient(RDDPG).It uses a recurrent neural network to memorize the relative position information between the robot and fruits and obstacles,and to infer hidden information such as the speed and movement of the robot,so as to accelerate deep deterministic policy gradient to learn how to avoid obstacles.Simulation results showed that the success rate of obstacle avoidance path planning of i APF was 100% with an average time of 3.6s;the success rate of RDDPG was 88.18% with a mean time of 50 ms.In other words,i APF is robust but less real-time,while RDDPG is real-time but less robust.(4)A guava harvesting robot platform was integrated,which consists of a six-degree-of-freedom robotic arm,two end-effectors,and a 3D visual perception and obstacle avoidance path planning software system.Based on this platform,in-filed harvesting experiments were performed to validate the effectiveness of the developed algorithms.Experimental results showed that:(i)in the absence of obstacle avoidance path planning,the success rate of guava harvesting with the pull-twist type end-effector was48.05% and the average running time was 25.9s;(ii)in the case of obstacle avoidance path planning by using RDDPG,the success rate of guava harvesting with the pull-twist type end-effector was 59.02% with an average running time of 20.4s;(iii)in the case of obstacle avoidance path planning by using i APF,the success rate of guava harvesting with the pneumatic shear end-effector was 80% with an average running time of 19.6s.These results not only reveal that obstacle avoidance path planning can significantly improve the success rate of robotic harvesting,but also confirm the effectiveness of the algorithms proposed in this paper.