| To address the requirements of the robot to search for the directional search task along the main stem in the tomato picking task,this paper conducts theoretical research,technical implementation and experimental verification on the experimental system design of the main stem tracking robot,multimodal image fusion instance segmentation,path planning and continuous tracking of the discrete visible area of the main stem.Relevant research can help the robot better realize the traversal of the fruits and stems around the main stem,and improve the picking efficiency and accuracy.The main content of the paper is as follows:(1)According to the characteristics of the tomato plant growth environment and the needs of the task,the main stem tracking robot test system was designed and constructed.In terms of hardware,the seven-degree-of-freedom Franka robotic manipulator is used as an actuator to achieve high-precision tracking performance.The visual perception mechanism includes a Basler ac A1300-60 gm NIR near-infrared spectral camera and an Intel RealSense D435i depth camera for multimodal recognition and 3D positioning functions.The visual perception mechanism includes Basler ac A1300-60 gm NIR near-infrared spectroscopy camera and Intel Real Sense D435i depth camera for multimodal recognition and 3D positioning functions.The mobile chassis is a self-built AGV mobile chassis,which can meet the flexible movement requirements of the robot in the tomato orchard.In terms of software,a distributed architecture is used to realize multi-computer communication and data exchange between the robotic manipulator and the industrial computer,effectively improving system reliability and scalability.(2)In view of the difference between the strong reflection characteristics of tomato main stems in specific bands and the similar-color background(leaves,green fruits),a multimodal image fusion method based on attention mechanism(YOLACTFusion)was proposed.In order to improve the recognition performance of RGB images,RGB images and NIR images with wavelengths ranging from 900 nm to 1100 nm are input at the same time,and the feature maps of different scales are weighted and fused using a parallel attention mechanism.In order to solve the prediction bias caused by pixel offset,a loss function for multimodal image information fusion is constructed.In order to reduce the computation and model size of the backbone network,local depthwise separable convolution is adopted and Conv-BN layers are merged.The experimental results show that,under the premise of maintaining the model’s high inference speed,YOLACTFusion improves the m AP of instance segmentation by 20.53% and reduces the number of parameters by 16.8%.The proposed method can effectively enhance the recognition performance of tomato main stems in similar-color background.(3)In order to solve the problems of poor guidance,high complexity and poor smoothness in traditional path planning algorithms,a time-optimal Rapidly-exploring Random Tree(TO-RRT)algorithm is proposed.Firstly,the attractive potential field and the repulsive potential field are established based on RRT to control the target bias probability of the random tree,and a node-first search strategy is introduced to reduce the number of failed growths.Then,the attractive step size and the step size dichotomy are introduced to improve the directional search ability of the random tree outside the repulsive potential field,while solving the problem of excessively large step size in extreme cases.Finally,the regression superposition algorithm is used to enhance the ability of the random tree to explore the unknown space in the repulsive potential field.The experimental results show that compared with RRT,the average running time of TO-RRT is shortened by 99.73%,and the average path length is shortened by 17.88%.The proposed method has high adaptability to different working environments and can effectively improve the planning efficiency of the manipulator.(4)To avoid interference from a variety of obstacles in the tomato cultivation environment that can affect robot tracking tasks,an obstacle avoidance tracking strategy based on the discrete visible area of the main stem is proposed.The attribution of the tomato plant’s sub-main stem is a crucial factor affecting the robot’s continuous tracking of the target.By binarizing the output mask and calculating the second-order moments of the image,the intercept and slope of the main stem in the pixel coordinate system are obtained.This allows for the determination of the affiliation relationship among all the main stems.Obstacle occlusion between adjacent sub-stems poses a challenge to the avoidance requirements of the robotic arm.The nearest centroid points of adjacent sub-stems on the same main stem are used as starting and ending points,and the path is replanned based on the difference between the depth value and the planning value.In order to overcome the tracking loss problem caused by multiple main stems existing in the current field of view at the same time,a task list is established based on the pixel distance from the elements of different main stems to the origin of the image coordinate system.According to the results of the unoccluded tracking test,the obstacle avoidance tracking test and the tracking test under the interference of multiple main stems,the target loss rate,the obstacle avoidance success rate and the accurate tracking rate are 0%,85% and 95% respectively.The proposed method achieves continuous tracking of discrete visible regions of the main stem without labeling other objects except the main stem. |
PDF Full Text Request