Research On Classification Recognition And Localization Of Ripe Citrus Fruits In Natural Scene

Ripe fruit harvesting is one of the primary tasks of fruit production operations.The development of an intelligent harvesting robot is conducive to shortening the picking time,reducing the cost of picking,and reducing the intensity of manual labor.The advancement of technology has important practical significance.In this paper,the citrus harvesting robot developed by the research group is used as an experimental platform.The aim is to improve the intelligence of the harvesting robot by studying the classification,recognition and positioning methods of ripe citrus fruits in natural scenes so that it has the ability of autonomous perception and harvest.The main research contents are as follows:1)Visual system construction and sample collection based on RGB-D camera.Based on the analysis and comparison of three existing depth cameras using different principles,combined with the actual requirements of the research,the advantages and disadvantages of binocular stereo vision,structured light and time of flight were comprehensively considered.Finally,a Kinect v2 depth camera based on the TOF principle was selected.The angle of view and depth error of the camera were analyzed,and the optimal working distance of the camera was obtained to ensure the accuracy and stability of the vision system.At the same time,the optimal working distance of the camera and the picking distance of the citrus harvesting robot determined the shooting distance range of the collected samples.Within this distance range,citrus images under different lighting angles are collected at different time periods to ensure that the training samples have good generalization.2)Construction and improvement of the classification and recognition model of ripe citrus fruits.Combined with the actual picking needs of the harvesting robot,different types of fruit were divided according to the growth and distribution of the fruit and the occlusion relationship.On this basis,there are 8 types of citrus training datasets were made,which include single fruit without occlusion,single fruit occluded by leaves,single fruit occluded by branches,single fruit occluded by leaves and branches and multiple fruits without occlusion,multiple fruits occluded by leaves,multiple fruits occluded by branches,and multiple fruits occluded by leaves and branches.At the same time,a YOLO v3 classification and recognition model with Darknet53 as the feature extraction network is constructed.Based on the analysis results,the network structure of the recognition model was improved accordingly,increasing the output of the network and improving the loss function of the network.The experimental results show that the average recognition accuracy of the improved model for different types of citrus fruits is 86.42%,which is 3.86% higher than the 82.56% before the improvement,and the detection speed is increased from 26.37 fps to 30.23 fps.3)Three-dimensional spatial positioning of different types of citrus fruits.The citrus fruits in different occlusion states are classified and recognized by the improved recognition model.The registration of color image and depth image is completed by using the mapping relationship between color image and depth image.On this basis,according to the transformation relationship between camera pixel coordinate system and world coordinate system,the three-dimensional coordinate information of citrus fruit picking point and the transverse and longitudinal diameter of fruit are obtained,so as to realize the three-dimensional spatial positioning of different types of citrus.The experimental results show that the average positioning error is 2.51 mm in the x-axis direction,2.71 mm in the y-axis direction,3.35 mm in the z-axis direction,2.43 mm in the horizontal diameter measurement and 2.41 mm in the vertical diameter measurement in the working range of the robot,which meets the actual picking demand of the robot.4)Development and application of citrus harvesting robot fruit target classification recognition and positioning system.Under Linux system,Qt5.10 is used as the software compilation platform,and related recognition and positioning algorithms are integrated in the Opencv4.0 image processing library.The development of the vision system and the design of the software interface are completed,and the interface operation of the vision system of the citrus harvesting robot is realized.The citrus harvesting robot equipped with the system is used to carry out multiple groups of picking experiments in the outdoor to verify the practicability and stability of the system.