Research On Recognition And Localization And Picking System For Kiwifruit Based On Deep Learning

With the continuous development of agricultural mechanization,using picking robots to harvest fruit will become a major trend.In kiwifruit production operations,fruit picking is an intensive task and intelligent picking can reduce labour costs,reduce manual labour intensity and improve picking efficiency,of which real-time identification and localisation of kiwifruit is one of the necessary conditions for intelligent picking.In this paper,kiwifruit datasets are constructed by collecting images of kiwifruit in natural environments and crawling web kiwifruit images for pre-processing,annotation and segmentation.Three models,Faster R-CNN,YOLO v4 and SSD-300,were used to train and test the kiwifruit dataset.According to the evaluation metrics in this paper,Faster R-CNN and SSD-300 have average recognition accuracy and speed for kiwifruit,while YOLO v4 shows greater advantages,but the backbone network CSPDarknet-53 of YOLO v4 is too computationally intensive leading to reduced detection speed,which cannot better meet the operational needs of kiwifruit picking robots.This paper proposes a lightweight convolutional neural network YOLO v4-GS based on the YOLO v4 model,replacing the CSPDarkent-53 backbone network with Ghost Net to reduce the computational effort of the model,introducing a cavity convolution layer to increase the sensory field and improve the detection accuracy of small targets.The SPP network was removed to reduce redundant computations.The experimental results show that the F1 value of the YOLO v4-GS model is 97%,the average detection accuracy is 98.90%,and the average detection speed is 11.96ms/frame.According to the model evaluation index of this paper,the YOLO v4-GS model has a good effect of light weight improvement.In order to objectively analyse the impact of the backbone network on the model performance and the overall performance of the YOLO v4-GS model,training trials were conducted for different backbone networks as well as comparing the YOLO v4-GS model with different deep learning models for analysis,and the YOLO v4-GS model showed significant advantages.Overall,the YOLO v4-GS model has excellent performance,with good recognition accuracy and speed as well as suitable model size,providing theoretical feasibility for a kiwifruit picking system that can identify kiwifruit quickly and accurately.In this paper,the kiwifruit picking system is designed and built in-house,with a ZED binocular stereo camera as the image acquisition tool,a Jetson Xavier NX system as the computational processing system,and an S6H4D＿Plus six-axis robotic arm as the actuator.Firstly,the camera calibration of the ZED binocular stereo camera is based on the principle of binocular vision to achieve spatial positioning of the kiwifruit,and secondly,the camera-arm hand-eye calibration is based on the ROS system to obtain the transformation matrix of the camera and arm coordinate systems to achieve spatial positioning of the kiwifruit in the arm coordinate system.To verify the accuracy of the positioning method in this paper,the average absolute errors in the three directions of x-axis,y-axis and z-axis were 1.42 mm,1.44 mm and0.67 mm respectively,and the average relative errors were 7.31%,7.17% and 3.41%respectively in the kiwifruit 3D coordinate verification tests under the vision system.In the kiwifruit picking system positioning verification tests,the average absolute errors in the x,y and z axes were 3.41 mm,3.40 mm and 3.43 mm respectively,with average relative errors of0.75%,2.19% and 0.41% respectively,which generally met the positioning requirements of the kiwifruit picking system in the actual picking operation.In the indoor simulated kiwifruit picking trials,the average time taken to pick a single kiwifruit was 11.72 s,with a picking success rate of 80.95%.In the kiwifruit picking trials in a natural environment,the kiwifruit identification accuracy was approximately 90%,the positioning accuracy was approximately95.6% and the picking success rate was approximately 90.69%.The overall evaluation success rate of the kiwifruit picking system was above 78%,the percentage of time taken for identification and positioning was approximately 29.03% and the percentage of time taken for the mechanical arm to move to the picking positioning point to pick a single kiwifruit was approximately 70.97%.Structural improvements were made based on hand claw problems during picking and better picking results were achieved in simulated indoor picking trials.In summary,the aim of this paper is to design and build a kiwifruit picking system by studying kiwifruit identification and positioning methods in natural environments,to achieve autonomous continuous kiwifruit picking functions,to improve the intelligence of the picking robot,and to provide a theoretical basis for agricultural mechanisation,which is important for promoting the development of agricultural picking robots.