Research On Dragon Fruit Position Recognition And Control System In Complex Picking Environment Based On Deep Learning

With the continuous development of smart agriculture,automated harvesting has become the main trend of fruit harvesting,and intelligent harvesting robots can reduce manual labor intensity,reduce costs,and improve harvesting efficiency.Real-time detection of fruits is one of the necessary conditions to realize automated harvesting by intelligent harvesting robots.In this paper,we construct a dragon fruit dataset by combining images collected in natural environment.Using the dataset,this paper compares and analyzes the detection effect of four one-stage target detection models including YOLOv5,YOLOX,YOLOv6 and YOLOv7 on dragon fruit under different light conditions,and finally choose YOLOv7 with higher accuracy and speed as the dragon fruit detection model,which has a mean precision of 85.7%,a mean recall of 88.7% and a mean average precision of 91.8%.Based on YOLOv7,this paper proposes a RDE-YOLOv7 to improve the detection accuracy of dragon fruit.By introducing Rep Ghost,decoupled head,and ECA attention mechanism into YOLOv7,RDE-YOLOv7 has a mean precision rate of 90.7%,a mean recall rate of 90.8%,and a mean average precision rate of 93.4%,which are respectively improved by 5.0%,2.1%,and 1.6% compared to the original YOLOv7.The dragon fruit detected by RDE-YOLOv7 will be divided into two categories: dragon fruit in the front view and dragon fruit in the side view,and the center coordinates of the dragon fruit will be obtained.Since picking the dragon fruit in the side view is more complicated,this paper proposes a dragon fruit posture estimation method to realize the side view dragon fruit picking.A semantic segmentation network is used to segment the dragon fruit,and then an ellipse fitting algorithm is used to obtain the two endpoints of the dragon fruit,then the trained Res Net is used to further divide the two endpoints into the head and the root,and finally the posture estimation is performed.This paper compares and analyzes the segmentation effect of three commonly used semantic segmentation models including FCN,UNet and PSPNet on dragon fruit,and finally choose the PSPNet model with 98.7% pixel accuracy and 90.3% mean intersection over union for dragon fruit segmentation,which takes 13 ms to detect an image on average.The average pixel error of the method is 39.8,the average angle error is 4.3°,and the accuracy of head and root classification is as high as 92%.Accurately obtaining the coordinates of the head and root of the dragon fruit can accurately evaluate the posture of the dragon fruit to guide the robotic arm to pick along a specific path.In this paper,we analyzed the working principle of several depth cameras,objectively compared the advantages and disadvantages between cameras according to the actual situation,and selected the ZED Mini binocular stereo camera.Secondly,the embedded system was compared and analyzed,and Jetson AGX Orin was selected as the model recognition and localization computational processing system in this paper.Finally,the six-axis robotic arm of the Six H4D＿Plus was selected as the actuator and the dragon fruit picking system was designed and built.In this paper,3D coordinates verification test,posture estimation and verification test,positioning verification test of picking system and dragon fruit picking test under natural environment were conducted.The test results show that the average absolute errors in the three directions of x-axis,y-axis and z-axis are 0.83 mm,0.49 mm and 0.32 mm respectively in the threedimensional coordinate verification test,which indicates that the internal and external parameters of the camera are more accurate after calibration in this chapter.In the posture estimation and verification experiments,the maximum cumulative errors of the dragon fruit and the three planes were 1.94°,4.24°,and 2.48°,respectively,and the small angle errors were able to achieve picking.The average absolute errors in the three directions of x-axis,y-axis and z-axis were 2.51 mm,2.43 mm and 1.84 mm respectively in the positioning verification test of the picking system,and the overall positioning errors could meet the positioning requirements of the dragon fruit picking system in the natural environment.In the dragon fruit picking test under the natural environment,the dragon fruit picking system constructed in this paper can pick the dragon fruit in the front view and the dragon fruit in the side view,which verifies the effectiveness of the identification and positioning method of the dragon fruit picking system in this paper and the feasibility of the whole picking system.