Research On The Methods Of On-tree Fruit Recognition Based On RGB-D Camera

The research and development of an automated fruit harvesting robot is an effective solution to solve the low efficiency of manual picking and reduce the cost of harvesting.The harvesting robot usually need to use visual system to obtain scene information,and with the help of image processing methods to detect and locate the fruit.The recognition rate of fruits is directly related to the success rate of subsequent picking.The traditional fruits recognition methods are mainly based on color images.Although it has achieved good results in certain situations,due to the complex and changeable environment in orchards,as the requirements for fruits detection continue to increase,the color and texture information provided by color image is limited,and it is difficult to meet the demand by only using color images.The RGB-D camera was used in this study,and the juicy peach was taken as object.On the basis of color features,the three-dimensional(3D)geometry features were fused to recognize fruits on trees,so as to provide a method for fruit detection in 3D space for fruit harvesting robots.The research work of this paper is as follows:1.Aiming at the problems such as fruits clustering and fruits blocking,an automatic fruit recognition method based on color and 3D contour information was proposed in this study.The point cloud data of the scene was acquired in a realistic orchard environment.After removing the background by a color threshold segmentation method,the pre-processed point cloud was divided into several point cloud clusters using the Euclidean clustering segmentation algorithm based on the distance and color difference.For clusters containing multiple fruits,the spherical segmentation method was applied to extract fruits one by one.Due to the limitation of camera view and actual occlusions,the obtained fruit point cloud is incomplete,these clusters are usually considered to contain only one fruit,and the circular segmentation method was exploited to improve the recognition result.The recall under three different degree of blocking environment such as no contact,fruits clustering,and fruits blocking reached 97.83%,89.74%,and 74.29%,respectively.The overall recall of fruit recognition under three situations was 88.68%,and the average detection time of a single fruit was about 292.4ms.The results showed that this study can provide theoretical guidance for fruit harvesting robots to detect fruits.2.It is found that the result of point cloud background segmentation will directly affect the subsequent fruit recognition.In order to improve the ability of fruit recognition for fruit harvesting robots in 3D space,a fruit point cloud segmentation method based on color and 3D geometry features was proposed in this study.Firstly,the candidate regions were obtained based on support vector machine(SVM)classifier by combining the color features and normal orientation features.Then the preliminary segmented color image and depth image were converted into 3D point cloud,and it was divided into multiple point cloud regions using the Euclidean clustering algorithm.The remaining non-fruit regions were eliminated by the viewpoint feature histogram(VFH)feature of each point cloud cluster to obtain the final segmented fruit point cloud.The experimental results showed that the segmentation accuracy of the fruit point cloud segmentation method was 98.99%,and the precision was 80.09%,which were both superior to the traditional color segmentation methods.In order to verify the influence of the segmentation method on fruit recognition,fruit detection methods based on shape analysis were exploited to process the segmented fruit point cloud,and the results showed that the segmentation method is more effective than the traditional color segmentation methods in improving the fruit recognition accuracy,and it can better meet the requirements of background segmentation in fruit recognition.3.In order to avoid too much reliance on human senses and experience for feature selection in fruit detection methods.In this study,the deep learning technology was applied to achieve the detection and counting of on-tree fruits.The fruit image data set was labelled manually,and the Faster R-CNN network was exploited to obtain the fruit detection model.Aiming at the potential misidentified targets,the detection results were mapped to the depth image,the distribution rule of pixel depth value in each detection box was analyzed,and the incorrect recognition box was eliminated based on the supervised learning method,so as to reduce the false detection rate of the model.The experimental results showed that the F1 score of the fruit detection model based on Faster R-CNN network was 88.41%,and the precision was only 83.74%.After removing the misidentified target by using the depth value variation rule in the detected box,the F1 score of fruit recognition reached 91.51%,the precision increased to 95.10%,and the recall was not affected significantly.The method can improve the accuracy of fruit recognition without significantly affecting the recognition rate,which provides a reference for removing the misidentified targets in deep learning methods.