Research And Implementation Of Mango Instance Segmentation And Detection Of Picking Point Based On Mask R-CNN

The goal of intelligent agriculture is to replace people with machines,thus liberating the productive forces and solving the disadvantages of traditional agriculture by improving production efficiency and reducing labor costs.Reliable and accurate multi-class segmentation is a key component of advanced robot perception tasks.In the context of agricultural robots,the image segmentation algorithm enables robots to understand the environment,pick crops,monitor crop health,and estimate yield.Aiming at the problems of shielding,overlapping and small targets in the process of mango segmentation and picking in the actual orchard scene,this paper studied the mango instance segmentation algorithm based on improved FCIS and improved Mask R-CNN,and the detection of mango picking point based on Mask R-CNN keypoints detection algorithm.Finally,in order to carry out the research,the mango instance segmentation and detection of picking point system is designed and developed,and the tasks of mango instance segmentation and detection of picking point on a portable laptop are realized.The main research contents and innovations of this paper are as follows:(1)Data set and model evaluation indexes.Data is the basis of deep learning algorithm,and the quality of data directly affects the performance of the trained model.The diversity of data enables the training model to cope with more situations and be more robust.In this paper,mango data were collected from natural orchard scenes,covering possible problems in the actual picking process,such as light change,background interference,especially fruit occlusion,overlap and small targets.In addition,the collected and sorted data are artificially enhanced and amplified,such as changing the contrast,brightness,color and sharpness of the image.In order to more accurately distinguish different size targets,this paper divides mango into three grades of large,medium and small according to pixel area.Labelme tool was used to manually label mango,and JSON files of training set,verification set and test set were obtained.In this paper,evaluation indexes of COCO data set were used to evaluate the performance of the model,and the comprehensiveness and accuracy of the evaluation indexes were analyzed.(2)Study on mango instance segmentation algorithm based on improved FCIS.FCIS is an instance segmentation algorithm based on position sensitive score map.Instead of adding full connection layer or convolution layer for each ROI to further extract features,foreground and background are separated directly by position sensitive map.In this paper,PSROIAlign is introduced to make the location of ROI pooling on the feature map more accurate and the final segmentation result more accurate.Feature pyramid structure(FPN)is introduced in basic convolution,and ROI is generated on feature maps of different sizes,so as to effectively segment mango individuals of different sizes.In order to further achieve the goal of lightweight,the Res50 of the basic convolution is replaced by the Mobile Net V2 structure,which achieves the model’s substantial compression and acceleration.The AP of the final model reached 70.10% with a size of only 47.00 M.(3)Study on mango instance segmentation algorithm based on improved Mask R-CNN.The Mask R-CNN algorithm achieves instance segmentation by adding a FCN branch on the basis of Faster R-CNN.In this paper,FPN-BU structure was built on the basis of FPN to further enhance the fusion of low-level features and high-level features,greatly shortening the fusion path of low-level feature map and high-level feature map.The experimental results prove that FPN-BU structure can improve the segmentation accuracy.In order to further improve the segmentation accuracy of overlapped fruits,Soft-NMS was used instead of NMS to avoid the error deletion of overlapped anchor frame,which enhanced the segmentation performance of the model in the case of overlapped fruits.In order to further improve the feature extraction ability of the prediction branch of the FCN mask and enhance the back propagation of the gradient,the four-layer ordinary convolution was replaced with the residual structure,which improved the final segmentation accuracy and made the network easy to train.All three improvements resulted in varying improvements in accuracy,eventually AP reached 83.00%.The effect on overlapping and occlusion data sets also proves the effectiveness of the improved strategy.The comparative experiment also shows the leading edge in the segmentation accuracy of the improved Mask R-CNN.(4)Detection of mango picking point based on Mask R-CNN.The traditional method to determine the picking point of fruit requires complex steps and manual intervention,and the final detection accuracy is not high and the robustness is poor.After studying the architecture of deep learning keypoints detection algorithm,this paper designed the automatic detection method of mango picking point based on the human keypoints detection algorithm of Mask R-CNN.Three keypoints(P1,P2,P3)were selected from mango and fruit stem to simulate human keypoints,and P1 is the real picking point.Annotation data were sent to the Mask R-CNN algorithm for training to predict the masks of keypoints,finally achieving automatic detection of mango picking points.The detection accuracy of 297 labeled picking points reached 92.90%,and the recall rate reached 96.97%.In addition,compared with the traditional picking point detection method,the accuracy is superior,but there are no complicated steps in the traditional method.(5)Design and implementation of mango instance segmentation and detection of picking point system.In this paper,a mango instance segmentation and detection of picking point system based on Windows was designed by using high-performance portable notebook,hikvision hemispherical network camera,hikvision hard disk video recorder and other hardware devices and encapsulating the trained model through Pycharm,Microsoft Visual Studio 2013 and other software.The system is composed of five parts: data acquisition,data decoding and preprocessing module,model selection and invocation module,automatic segmentation and detection module,and result display module.