Multi-cluster Kiwifruit Image Detection In Field Using Deep Learning

It is a challenge for all-day automatic detection of kiwifruit in field because of the daily illumination variations and high color similarity between kiwifruit and complex background of leaves,branches,and stems.Especially,kiwifruits are growing in cluster,which makes the fruits occluded,overlapped and adj acent to each other.In order to achieve the goal of fast and accurate detection kiwifruits under the conditions mentioned above,this study proposed different algorithms.The main research contents and conclusions are as follows:(1)The construction of the kiwifruit dataset was studied.The method of capturing images used in this study was based on placing the camera underneath the fruits,with its central axis perpendicular to the canopy.The images were taken in different times of the all-day with varied lighting conditions,two images(with or without flash)were acquired in the morning and afternoon respectively.For the night,two images were acquired with a LED illumination or with flash.In order to achieve robust detection of kiwifruit in the field,this study takes into account all kinds of interference that may occur when detecting fruits.Through luminance transformation,motion blur,contrast enhancement and reductionand adding Gaussian noise to enrich image training set.Ground truth data for network training and testing was created using manual labeling(using rectangular bounding boxes)and interactive markup(manual interception)of the fruits on all the dataset images.(2)Multi-cluster kiwifruit image detection in field based on LeNet convolutional neural networks.For the study of multi-cluster kiwi detection during the day,the LeNet was used for feature learning.The RGB image of kiwifruit was transferred into a matrix with the size of 32×32 as the input of the network,stochastic gradient descent was used to train our models.In addition,the CNN took advantages of the part connections,the weight sharing and Max pooling techniques to lower complexity and improve the training performance of the model simultaneously.The network used rectified linear units(ReLU)as activation function and batch normalization(BN)method to optimize the CNN architecture,which could greatly accelerate network convergence.The results showed that the overall detection rate of the model reached 89.29%,and it only took 270 ms in average to recognize a fruit.However,some fruits were wrongly detected and undetected.(3)Multi-cluster kiwifruit image detection in field based on Faster R-CNN.For the study of multi-cluster kiwifruit detection in all-day,through the performance comparison tests of the current popular deep learning framework,and Caffe was selected as the framework.The study used the method of migration learning to initialize the network,optimized the model according to the target information,and trained ZF network and VGG16 network with the training samples.The results showed The Faster R-CNN with VGG16 model takes an average of 347 ms to measure each image and a model size of 512 MB.It was 77 ms slower than the average detection speed of the Faster R-CNNwith ZF model(270 ms),and the model as a whole is relatively large,making it difficult to migrate to small mobile devices.However,the average detection accuracy of the Faster R-CNNwith VGG16 model for the test set is 87.61%,which is higher than 72.50%of the Faster R-CNN with ZF model.(4)For all-day kiwifruits detection research,the study proposes a fast and precision detection of kiwifruit in field using improved YOLOv3-tiny model.According to the characteristics of kiwifruit images,by introducing deep convolutional networks into YOLOv3-tiny model,and through higher resolution inputs,multi-scale strategy and fine-tuning to develop a deep YOLOv3-tiny(DY3 TNet)model.Comparing with ZF,VGG16,YOLOv2,and YOLOv3-tiny,the DY3TNet model obtained the highest average precision of 90.05%,but with the smallest weight of 27 MB.It only cost 34 ms on average to process an image with the resolution of 2352x 1568 pixels.In addition,the DY3 TNet model and the YOLOv3-tiny model showed better performance on images with flash than that without flash.It can be concluded that the flash is positive on kiwifruit image detection.Moreover,the experiments indicated that the image augmentation process could improve the detection performance for the DY3 TNet model,and a simple and consistent lighting condition can improve the success rate of detection in field.Experimental results demonstrated that the improved model DY3TNet model is smaller and more efficient and robust.To sum up,this paper proposed a feasible solution for all-day kiwifruit automatic detection in field,and good experimental results of the different kiwifruit testing datasets had been obtained.In this paper,the DY3 TNet model ameliorated,based on YOLOv3-tiny model,improved the recognition accuracy and ensured the realtime,and implemented miniaturization The study provides a new thought for the detecting kiwifruit in field in real time and accurately,and provides strong support for the research on crop-load estimation and robotic picking kiwifruit with multi-arm operations.