Research On Online Sorting Method Of Citrus With External Defects Based On Machine Vision

Online sorting of citrus fruits with external defects is an important task in the processing line,which can avoid further infection of health fruits.Sorting fruits with external defects is mainly manually conducted at the moment,which is time-consuming,laborious,and inaccurate.At the same time,different types of defects will induce different symptoms on the rind,making it difficult to develop automatic non-destructive detection methods.To address these issues,this paper explores the detection,tracking,and trajectory prediction methods of citrus fruits with external defects using machine vision and deep learning,and they can be combined to realize online sorting on the processing line.The main contents of this paper are as follows:(1)Since recognizing and locating the fruits with external defects is difficult in a single frame,we first developed a Convolutional Neural Network(CNN)-based method to accurately detect the defective ones.Three detection algorithms with strong feature extraction capacity,including SSD,YOLO-V4,and Efficient Det,were compared for the performance to detect defective fruits.Videos were taken when the fruits rolling on a real processing line,and images were extracted from the videos and then used as the training,verification,and evaluation dataset.The experimental results showed that YOLO-V4 was the most suitable model considering its high computation efficiency,with the final F1 score of 88.3% and average processing time of a single frame of 69.9 ms.However,as the processing time of a single frame should be less than 40.0 ms to realize on-line sorting,the computation efficiency of the optimal model was still not satisfactory.(2)To further improve the computation efficiency to realize online sorting on the processing line,YOLO-V4 algorithm was further customized to be more lightweight.Fristly,a variety of lightweight networks,such as Mobilenet-V1,Mobilenet-V2,Mobilenet-V3,Ghostnet,and Densenet121,were used to replace the backbone CSPDarknet53 in YOLO-V4 for feature extraction,and the performances were compared to obtain the preliminary optimal model Mobilenet-V2-YOLO-V4.After that,the PANet module,SPP module,and YOLO head module in the Mobilenet-V2-YOLO-V4 were replaced and pruned,and the parameter optimization test was carried out,resulting the customized optimal detection model Mobile-Citrus.The experimental results showed that the F1 score of the model was 84.5%,and the average processing time of a single frame was 12.3ms.As a result,the customized optimal model had significantly improved computation efficiency but maintained the detection accuracy,and it also achieved good real-time performance on the processing line.(3)As the fruits would present different surfaces during rotation on the conveyor,we developed a tracking algorithm to track the fruits in continuous frames to identify their true types.Combined with the customized optimal detection model,a variety of tracking algorithms,including Kalman-Tracker,LSTM-Tracker,and Transformer-Tracker,were adopted to realize online citrus tracking.The experimental results showed that the tracking performance of LSTM-Tracker,was the best,with the multiple object tracking accuracy MOTA of 99.4%,and multiple object tracking precision MOTP of 89.7%,with the average processing time of a single frame was 15.3ms.Although considering the processing time of the optimal detection model,it still meets the real-time requirements of production line sorting.(4)Using robotic grippers to pick the defective fruits out on the conveyor is a key step for citrus sorting,and it is necessary to predict the future path of the fruits to guide the grasping.Trajectory prediction methods based on Kalman,LSTM,and Transformer were developed,and tests to verify and evaluate the method performance were carried out.The results showed that the trajectory prediction errors of the three trajectory prediction algorithms were small,which can be used to predict the trajectory of defective fruits and guide the manipulator for sorting.However,when considering the processing time of detection and tracking,only the trajectory prediction algorithms based on Kalman and LSTM can meet the real-time requirements of the processing line.Among them,the trajectory prediction algorithm based on Kalman had the best performance.The average absolute error of trajectory prediction was 3.86 pixels,accounting for about 6.4% of fruit diameter,and the average processing time of a single frame was 0.235 ms.(5)In order to obtain the sorting algorithm of the processing line combing the functions of detection,tracking,and trajectory prediction,three sorting methods,including Mobile-Kalman-Citrus,Mobile-LSTM-Citrus,and Mobile-Transformer-Citrus,were designed based Kalman,LSTM,and Tansformer,respectively.Considering the real-time and classification accuracy requirements of sorting defective fuirts on the processing line,the tracking performance of the Mobile-LSTM-Citrus was the best,with the MOTA of 99.4%,MOTP of 89.7%,the average absolute error of trajectory prediction of 4.33 pixels,the average classification accuracy of 93.7%,and the average processing time of a single frame image of 35.3ms,satisfying the efficiency and accuracy requirements of online sorting on the citrus processing line.