The Method And Test Of Orange Locating And Picking Based On Machine Vision

As one of the important bulk agricultural production,orange(Citrus sinensis)is usually planted in hills and mountains,and requires a great amount of labor to harvest.For this reason,it is urgent to realize mechanized and intelligent picking of orange.In response to the complex growth environment,light interference,branch and leaf occlusion and other factors that influence the operation of the orange picking actuator,in this work,locating and picking methods of orange based on machine vision were studied,and validation test was conducted,which provided theoretical and technical references for the development of fruit and vegetable picking robots.The main research contents were as follows:(1)The design and construction of vision system of the orange picking actuator:aiming at the three-dimensional locating requirements of the fruit,binocular structured light depth camera Real Sense d435 i was selected to build the vision system,and installed by the eye fixation method.Based on image processing algorithms combined with the depth camera,three-dimensional locating of fruit was completed.Then,an calibration experiment was carried out so that the camera’s internal and external parameters could be achieved.After calibration,the coordinate system conversion of the three-dimensional coordinates of the fruit could be completed.After that,based on the principle of the shortest path,the optimal picking order of fruits was determined.Finally,by using Py Qt software,a human-computer interaction interface was designed to improve the efficiency and convenience of operation.(2)The comparison,selection and optimization of orange recognition algorithms:three methods were used to detect the fruit,namely,the image processing algorithm based on morphology,the 3D point cloud processing algorithm,and the deep learning model.In the image processing algorithm based on morphology,the fruit was segmented and extracted based on color difference method combined with Otsu algorithm,then the contour of the segmented fruit was detected to obtain the centroid coordinates of the fruit.Especially,for overlapping fruits,a method based on the watershed algorithm combined with morphological processing algorithm was proposed to segment overlapping connected regions.In the 3D point cloud processing algorithm,a threshold segmentation method on the foundation of color difference was used to separate the fruits and remove the background.Additionally,a clustering method based on euclidean distance and RANSAC algorithm was used to classify and fit multiple fruits to obtain the centroid location and radius of the fruits.In the deep learning model,an improved lightweight YOLOv4 model which used the Mobile Net v2 module to replace the CSPDarknet network as the backbone network and replaced standard convolutions by depthwise separable convolutions was proposed.It was shown that the improved model achieved a recognition precision of 97.57%,recall of 92.27%,F1 of94.85%,and average precision of 97.24%,and the size of model was 46.5 M.For the sake of verifying the effectiveness of the three recognition algorithms and select the best recognition algorithm,a comparative experiment was conducted: the above three recognition algorithms were validated in the same test set.The results demonstrated that the recognition success rates of the image processing algorithm based on morphology,3D point cloud processing algorithm,and improved YOLOv4 algorithm were 79.49%,74.36%,and 98.72%,respectively.Obviously,the improved YOLOv4 model achieved the best recognition success rate compared to the other two algorithms,and had strong robustness in complex natural environments.(3)The design and validation of the orange locating algorithm: based on depth camera combined with improved YOLOv4 model,an orange locating method was proposed.This method decomposed the fruit locating process into two parts: twodimensional locating and depth acquisition.After collecting the color map and depth map of the fruit image,the acquisition of the two-dimensional coordinates of the centroid of the fruit was completed through the improved YOLOv4 model.By recording the location of the centroid point and mapping it to the depth map obtained by the depth camera,the depth value and the fruit radius could be obtained and calculated.The above locating algorithm was validated in the test set.The results showed that the recognition success rate had reached 98.72%,and the threedimensional locating success rate had reached 96.15%.Besides,the MAE was 3.48 cm,and the MAPE was 2.72% in the process of depth acquisition.(4)The construction and testing of hardware and control system of the picking actuator: the picking actuator contained the below components: a rectangular coordinate manipulator was used as the motion actuator,a three-claw flexible adaptive mechanical claw was used as the picking end effector,a laptop was used as the upper computer,and the STM32 was used as the main controller.After the three-dimensional locating of the fruit was completed,the task of approaching the target fruit was carried out by STM32 one-chip computer by outputting a fixed number of pulses based on timers to control the fixed number of turns of the stepper motor of each axis.Then,based on RS485 communication principle,the mechanical claw was controlled to complete the fruit grasping and releasing.(5)The design and implementation of the locating accuracy experiment and picking experiments: to explore the accuracy of the proposed method,a locating accuracy experiment was carried out under indoor conditions,and the results showed that the MAE in the X,Y,and Z directions were 5.85 mm,6.76 mm,and 8.59 mm,respectively,within the fault tolerance range of the end effector,meeting the locating accuracy requirements during actual picking work.Additionally,in indoor and outdoor environments,picking experiments were conducted.The results turned out to be that the picking success rate of the algorithm proposed had reached 87.29% in indoor environments Before the outdoor picking experiment,considering the failure of depth information acquisition caused by outdoor lighting and the effective travel of the robot arm,a fruit picking space was set for outdoor environments.The fruit could be picked only when the fruit was within the picking space.The outdoor picking experiment indicated that the success rate of picking under outdoor conditions was 79.81%,and the average picking time of a single fruit was 8.9 seconds.