Citrus Fruit And Branch Segmentation In An Orchard Environment Based On Deep Learning

As the garden fruit with the largest planted area and production in China,citrus addresses a large market demand,but the picking process still relies on manual work,which is labour-intensive and inefficient.Therefore,the use of picking robots instead of manual picking can effectively improve productivity,reduce the work intensity of practitioners and meet the future needs of smart agriculture development.As the core component of a picking robot,the vision system is of great significance to the development of a picking robot.In order to avoid collisions between picking robots and branches during harvesting in complex orchard environments,this study proposes a method for segmenting and localising citrus fruits and branches,improving the accuracy and speed of segmenting fruits and branches,reducing the rate of missing small target fruits,and combining the generated point cloud data to achieve fruit and branch localisation,aims to provide visual guidance to the citrus picking robot for obstacle avoidance and picking.The main research work is as follows:(1)The production of citrus fruit and branch dataset in natural environment.citrus fruit and branch image samples were collected at Guangxi Citrus Research Institute,citrus fruit features include branch shading,overlapping and other states,and they were labeled according to the growth characteristics of citrus and branches.In order to improve the generalization ability of the model,data enhancement strategies to enrich the dataset,and finally 1800 images are obtained as the dataset of this study to prepare for the subsequent training of the network model and algorithm analysis and validation.(2)Selection of deep learning network models.The current YOLACT algorithm,SOLO algorithm and Mask RCNN algorithm,which are commonly used in deep learning in the field of object segmentation,are analysed in experiments for this research dataset,and based on Mask RCNN segmentation network model for citrus fruits and branches is used to build a segmentation network model to ensure the segmentation capability of the model.(3)A citrus fruit and branch segmentation model based on improved Mask RCNN.To further improve the accuracy and speed of Mask RCNN for accurate segmentation of citrus fruits in natural environments,a Mask RCNN instance segmentation network model based on Mask RCNN is used and improved,using MobileNet v3 instead of ResNet101 as a feature extraction network,and the depth separable convolution is used instead of normal convolution in this structure to reduce the amount of model operations.In addition,an improved BiFPN is introduced into the CARAFE upsampling module to replace the FPN structure to enhance feature fusion,improve the segmentation accuracy of citrus fruits and branches in natural environments.The experimental results show that the improved Mask RCNN network model achieves 93.74%,86.93% and 87.99% accuracy,recall and AP for fruit segmentation,and 95.58%,85.84% and 92.07% accuracy,recall and AP for branch segmentation,respectively,in the complex orchard environment,which is higher than the original model on fruit and branch The AP on segmentation was improved by 2.38% and1.12%,respectively,and the number of parameters was reduced by 32.23%.The processing speed of a single image on the computer was 0.184 s,and the detection speed was improved by 43.73%.The improved model has improved the precision of citrus fruit segmentation for the occurrence of occlusion and small target citrus fruits,and the number of model parameters has been significantly reduced to achieve fast and accurate segmentation of citrus fruits and branches under natural environment.(4)Three-dimensional spatial fitting and localization of citrus fruits and branches.The internal parameters of the camera are obtained by calibrating the Azure Kinect DK camera,and the conversion relationship of coordinate system is derived.After the RGB images acquired by Azure Kinect DK are detected,the depth images of the corresponding area is incorporated to generate the initial target point cloud,and the RANSAC algorithm is used to fit instance object generated using the sphere and cylinder models respectively to determine the 3D spatial location of the fruit and branch.The test results showed that the average positioning error of citrus fruit in the X-axis direction was 3.38±1.10 mm,in the Y-axis direction was 2.98±1.34 mm,in the Z-axis direction was 3.11±1.21 mm,and the radius error was 3.65±1.23 mm by the positioning method of this study.The average positioning error was 10.82±3.64 mm in the X-axis direction,12.36±3.44 mm in the Y-axis direction,8.14±2.48 mm in the Z-axis direction,and 4.50±1.63 mm in the branch radius.It meets the positioning error requirements of picking robots and provides stable and reliable technical support for obstacle-avoidance picking in orchard environments.