Fruit Recognition And Yield Estimation Based On 3D Point Cloud

Fruit growth monitoring and yield estimation at harvest require a lot of manpower.The methods based on machine vision can effectively improve fruit planting efficiency.The information of two-dimensional（2D）image is limited,and the methods based on a single 2D image can only realize fruit identification from a single perspective.In order to meet the requirements of more accurate fruit identification and yield estimation in three-dimensional（3D）space,this paper introduced 3D point cloud data into fruit recognition.The point cloud registration algorithm was used to obtain the information of the whole plant.The fruits were recognized by combining color features and three-dimensional geometry features,so as to realize the fruit counting and yield estimation of the whole plant.The research content of this paper is as follows:1.Aiming at the problems of limited information in a single image and difficult segmentation of green fruit,a fruit recognition and counting method based on 3D point cloud registration was proposed.Firstly,an RGB-D camera was used to collect the multi-angle point cloud data of plants.The background and noises were removed.Then,the random sampling consistency algorithm was used to fit the cylinder to obtain the parameters of the rotating central axis,and the point cloud was rotated at a fixed angle around the central axis to complete the initial registration.After that,the point-to-plane iterative closest point（ICP）algorithm was used to complete the precise registration to obtain the complete point cloud.Finally,Euclidian clustering was used for point cloud segmentation,and the random sample consensus（RANSAC）algorithm was used for spherical segmentation of the clustered point cloud to obtain the 3D spatial position of each fruit and count.In this study,nine potted kumquat plants（149 fruits in total）were identified in the fruit growing stage.The results showed that the total recall was 85.91%,precision was 79.01%and ₁F score was 82.32%.Compared with the ground truth,the coefficient of determination and mean absolute percentage error of the number of fruits calculated by the proposed method were 0.97 and16.02%.The experimental results showed that the proposed method was independent of color information and could effectively immature green fruits in the whole plant,which could provide a reference for fruit identification and yield estimation.2.Aiming at the problem of wrong recognition caused by background information such as leaves in fruit recognition,a fruit point cloud segmentation method based on the super-voxel segmentation method was proposed.Firstly,the super-voxel blocks of the point cloud were obtained by the super-voxel segmentation algorithm based on color,normal vector and distance information.The color and 3D geometric features of the super-voxel blocks in the fruit area and the non-fruit area were extracted respectively.And then,the features were sent into the least square support vector machine（LS-SVM）classifier for training.Finally,the trained classifier was used to remove the background point cloud and the fruit point cloud segmentation was realized.After removing the leaf background,the fruit point cloud registration and fruit recognition were carried out.The results showed that the total recall was92.62%,precision was 87.90%and ₁F score was 90.20%.The recognition accuracy was significantly improved compared with that before removing the background.Compared with the ground truth,the coefficient of determination was 0.99,and the mean absolute percentage error was 8.57%.The experimental results showed that the method could effectively remove the background point cloud and improved the fruit recognition accuracy when the fruit was closely connected with the branched and leaves.3.For plant point cloud registration,the method of using external objects to assist registration has some limitations.To expand the application scenarios of the fruit recognition algorithm,a new initial point cloud registration method was proposed.Firstly,the RGB-D camera was used to obtain multi-angle point clouds.The normal distribution transformation（NDT）algorithm was used for initial registration of the plant point clouds.Then,the spherical fitting algorithm was used to calculate the spherical center of fruit in the plant,and further registration was performed according to the spherical center coordinates.Finally,the point-to-plane ICP algorithm was used to achieve accurate registration.The experimental results showed that the root mean square error（RMSE）of overlapping point cloud after registration was 0.186cm,which was lower than the registration method using feature matching,NDT and ICP binding method.In order to verify the effectiveness of the method for fruit recognition,the fruit recognition method based on spherical fitting was applied to the registered point cloud.The total recall was 85.32%,precision was 76.23%and ₁F score was80.52%.The method can realize plant point cloud registration independently of external objects,which can provide a reference for fruit recognition and yield estimation in outdoor scenes.