Detection And Localization Of Field Apples Using Kinect

Apple is one of the large-scale agricultural products grown in China,which is an important pillar of economic development.However,the picking of apples still depends on labor force,which leads to high costs and low productivity.Therefore,it is of great significance to develop a high-level picking robot to take place of manual operations.In the field environment,a low-cost Kinect sensor was employed to acquire jazz apple RGB(Red,Green,Blue)image,depth information and point cloud in wall-planting mode.Firstly,the convolution neural network was employed to detect apple fruits with the original RGB image.However,branch-occluded fruit was easily divided into several separate parts,which may cause fruits to be misdetected as several fruits.In order to determine the fruit picking order of overlapped fruit,multi-class samples of the apple image were manually labeled.And then in order to further improve the detection accuracy,the new network model used parameter migration was employed to detect apple fruits with the background removed RGB image.Finally,point cloud was employed to locate apple fruits.The specific contents are as following:(1)Apple image detection using the original RGB image.Firstly,the data was augmented by the geometric transformation and image enhancement method.And then,the single and multi-class samples of the apple image were manually labeled.After that,the data augmented sample was mapped to the tag file.Finally,two common Faster-RCNN(Regions Convolutional Neural Network)based deep learning pre-trained models(ZF and VGG16)were employed to detect apple fruits with the original RGB image.The experimental results showed that VGG16 network model had better performance for apple detection,in the single-class detection model,the AP(Average Precision)of fruit was 88.12%,and the detection time was 0.182s/frame;in the multi-class detection model,the AP of unoccluded fruit was 90.90%,the AP of leaf-occluded fruit was 89.94%,the AP of branch-occluded fruit was 85.82%,the AP of overlapped fruit was 84.82%,the mAP(mean Average Precision)was 87.87%,and the detection time was 0.241 s/frame.(2)Apple image detection using background removed RGB image.Firstly,the background objects was removed effectively using the depth information with depth values exceeding 200 cm.And the image data was augmented and the corresponding tag file was mapped.Then the new VGG16 network model using parameter migration was employed to detect apple fruits with the background removed RGB image.The experimental results showed that,in the single-class detection model,the AP of fruit was 89.32%,which was 1.2%higher than that of the original RGB image,and the detection time was 0.181s/frame;in the multi-class detection model,the AP of unoccluded fruit was 91.12%,the AP of leaf-occluded fruit was 90.25%,the AP of branch-occluded fruit was 87.34%,the AP of overlapped fruit was 86.48%,the mAP was 88.80%,which was 0.22%、0.31%、1.52%、1.66%and 0.93%higher than that of original RGB image respectively,and the detection time was 0.240 s/frame.(3)Apple localization of 3D(3-Dimensional)space using point cloud.Firstly,two pixel coordinates(top-left and bottom-right)on the bounding box was extracted using MATLAB and the average value was calculated as the coordinates of the center point.Then sequentially the coordinates of the top,bottom,left,right,top-left,bottom-left,top-right and bottom-right adjacent points of the center point was extracted.What s more,the background,outliers and internal high frequency noise of point cloud were removed using the through filtering,statistical filtering and bilateral filtering respectively.Finally,the 3D spatial coordinates of the 9 points were extracted from point cloud and the average value was calculated as the coordinates of the final grasping point of apple.Two common Faster-RCNN-based deep learning pre-trained models(ZF and VGG16)were employed to detect the single and multi-class samples of apple fruits with the original RGB image.VGG16 network model had better performance for apple detection than ZF network model in the field environment.Then the new VGG16 network model was employed to detect apple fruits with the background removed RGB image,the AP was further improved.Finally,the 3D coordinates of apples were extracted from point cloud and the picking order of apples were determined.This paper provides some methods for the target detection and location of the apple picking,and it has obtained some conclusions for reference.