Fruit Detection And Pose Estimation Based On RGB-D Camera

The picking action of the fruit picking robot relies on the detection and pose estimation of the fruit by its visual inspection system.In the complex natural scene,due to the problem of low recognition accuracy and poor robustness in detection and recognition caused by RGB images,the point cloud images collected by RGB-D camera were used in the field of fruit peduncle detection and pose estimation.Theories of machine learning(partial least squares discriminant analysis,subtractive clustering,etc.)and algorithms in 3D point cloud processing(fast point feature histogram,sparse outlier removal,etc.)were used to provide accurate automatic picking of fruit detection and pose estimation.The main work of the thesis is as follows:1.It is common for fruits to be occluded in nature,and many fruits(such as sweet peppers)are irregular,so the success rate based on the fruit shape detection method in the previous study will therefore decrease.In order to overcome the influence of fruit occlusion,a method for detecting and counting fruits was proposed,which provided scientific and reliable technical guidance for automatic picking robots.First,the fruit region was segmented from the point cloud of the tree by applying the color threshold of R-G.Then,the sparse outliers and the noise in the fruit cloud were removed.Finally,a point cloud for each fruit was detected and counted based on a subtractive clustering algorithm.The algorithm was tested in the sweet pepper point cloud dataset.For all fruits that were at least partially visible in the scene,the true positive rate was 90.69% and the false positive rate was 6.97%.Undetected fruits indicate that these are highly enclosed fruits with only a small portion of the surface visible.The results show that this study can overcome the influence of fruit occlusion on automatic detection to some extent.2.There are two types of end effectors for the separation of fruits,one based on the cutting of the fruit peduncle detection and the other based on the pose estimation.This study proposed a visual detection method for the peduncle detection of sweet pepper,providing guidance for cutting end effectors.The color and geometric information obtained from the sweet pepper point cloud and the supervised learning method were used to perform the fruit peduncle detection task.First,the entire sweet pepper point cloud was subdivided into pulp and fruit peduncles.An improved 3D descriptor consisting of hue,saturation,value(HSV)features and fast point feature histogram(FPFH)features were then extracted from the pre-processed point cloud,respectively.Finally,the partial least squares discriminant analysis(PLSDA)classification model was developed using HSV+FPFH descriptor,and the 88.96%average classification accuracy of the challenging sweet pepper point cloud dataset was obtained,which was better than the 83.60% based on HSV descriptor.Compared with the HSV descriptor,the HSV+FPFH descriptor proposed in this study can better meet the accuracy requirements of the automatic picking system.3.In natural scenes,the fruit peduncles are usually thin and usually occluded,so the application of cutting end effectors based on fruit peduncle detection is more limited.Thisstudy proposed a method for fruit pose estimation that provides guidance for grasping end effectors.In this method,the normal of the local plane at each point in the fruit point cloud was first calculated,and the point cloud was divided by several candidate planes.Then a scoring strategy was used to calculate the score of each plane respectively,and the plane with the lowest score was selected as the symmetry plane of the point cloud.Finally,the axis of symmetry can be calculated from the selected plane of symmetry,and the pose of the fruit in the space can be obtained by using the axis of symmetry.The performance of the proposed method was evaluated by simulating point cloud and sweet pepper cloud dataset.In the simulation data test,the calculated average angular error between the symmetry and the real axis was about 6.5°.In the sweet pepper point cloud dataset test,when the fruit peduncle was removed,the average error was about 7.4°,and when the peduncle of the sweet pepper was complete,the average error was about 6.9°.These results indicate that the method is applicable to the pose estimation of sweet pepper and can be adjusted for other fruits and vegetables,and further guides the grasping operation.