Research On Image Recognition Methods For Multifunctional Fruit And Vegetable Harvesting Robot

The mechanized and intelligent cultivation is the inevitable trend of agricultural development.Fruit and vegetable harvesting is an important part of agricultural production.How to realize automatic and unmanned fruit harvesting is one of the important research focuses in the field of agricultural engineering in recent years.However,most of the developed harvesting robots can only work for a certain kind of fruit.They are idle in most of the year,which increases the cost and is not conducive to commercial promotion.Therefore,a robot that can harvest a variety of common fruits has important practical value and commercial prospects.As the eyes of the harvesting robot,the vision system is the key to recognize and locate various kinds of fruits.In this study,the apples and cucumbers are taken as the main research objects for their representative color and shape features.The study focuses on image recognition methods of fruits.Then,the related methods are applied to recognize and locate other kinds of fruits.According to the characteristics of different fruits,3 different methods for fruit recognition and location are proposed,which are mainly based on color feature,shape feature and DCNN(Deep convolutional neural network)respectively.This research mainly includes the following 4 aspects:(1)In view of the color difference between mature apples and the background of orchards,an algorithm based on color features is proposed.The color factors that can reflect the color difference are combined into the multi-channel image.Then,the sparse convolution kernel that can take pixel information in the neighborhood into consideration is proposed to segment multichannel images.To determine the elements in the sparse convolution kernel,its template is used to collect neighborhood sample data.Then the data is input into a linear classifier to train a classification model.Finally,the coefficients of the classification model are converted into the sparse convolution kernel.The algorithm combines the classification method with convolution operation to realize fruit recognition and location.The sparse convolution kernel samples pixels in the neighborhood at intervals.Compared with the segmentation effect of the common convolution kernel with the same size,the sparse convolution kernel not only has the same segmentation effect but also can reduce the repeated operation in the convolution process significantly and improve the operation speed.The results of the experiment show that the precision of the algorithm reaches 91.12%,its recall reaches 86.67% and the average location error is 8.12%.(2)The color of apple fruits is distributed unevenly and easy to be affected by illuminations,but the shape of fruits is not sensitive to light.In view of these,an algorithm based on shape feature is proposed.Firstly,the candidate regions are recognized based on color and texture features.Then,the fruits are further recognized and located in the candidate regions by the shape feature of fruits.To recognize candidate regions,the image is divided into a series of super-pixel blocks and then these blocks are classified into fruit blocks and non-fruit blocks by BP(Back propagation)neural network.In the candidate regions,the multi-scale sliding windows are used to traverse and the HOG(Histogram of oriented gradient)operator is used to describe the shape of the objects in the sliding windows.Then the feature vectors of HOG are classified by an SVM(Support vector machine)classifier to determine whether the object in the corresponding sliding window is a fruit.The shape feature of fruits is the key and the color and texture features just play an auxiliary role to narrow the detection range of the shape descriptor.The results of the experiment show that the precision of the algorithm reaches 95.12%,its recall reaches 89.80% and the average location error is 7.26%.(3)Aiming at the green color and the narrow shape of cucumber fruits,the Mask RCNN(Region-based convolutional neural network),an instance segmentation framework based on DCNN,is proposed to recognize and locate cucumber fruits.Compared with the methods that recognize and locate fruits with rectangular bounding boxes,the Mask RCNN can segment objects in pixel level which can locate cucumber fruits with higher precision in the horizontal direction.To improve the accuracy of recognition and location further,the structure of the framework and the parameters of anchors are customized according to the color and shape characteristics of cucumber fruits.The uncertainty of convolution features in the process of training and learning is compensated by incorporating artificial features into the framework and parameters.The experiment shows that the improved structure of the framework is more effective and in line with the recognition of cucumber fruits,which not only improves the accuracy of recognition and location but also improves the operational efficiency of the algorithm.The precision of the algorithm reaches 90.68%,its recall reaches 88.29% and the average location deviation is 3.58 pixels.(4)The algorithms of fruit recognition based on color feature,shape feature and DCNN are applied to mature red tomatoes,immature green tomatoes and green peppers respectively.The relevant parameters in the corresponding algorithms are modified according to the characteristics of different fruits.Meanwhile,the training sets and test sets of different fruits are established to train and test the corresponding classification models in the corresponding algorithms.According to the experimental results,the application objects and adaptive scopes of different algorithms are analyzed and summarized.In the above researches,the main innovation points of the corresponding algorithms are the structure design and application method of the sparse convolution kernel,the application method of the candidate region recognition and the fusion method of artificial features and DCNN respectively.The 3 different algorithms can not only recognize apples,cucumbers,tomatoes and green peppers but also be extended to recognize other fruits with similar features.Various fruit recognition algorithms provide strong support for multifunctional fruit and vegetable harvesting robots so that the robots can select an appropriate algorithm for different fruits.