Recognition And Remote Growth Parameter Detection Of Apples On Trees During Whole Growth Period

In the process of apple growth and development,it is of great significance to acquire the growth parameters of fruit in time for studying the growth rules of fruit and guiding fruit growers to conduct scientific management.Traditional apple fruit growth parameters are mainly obtained by manual measurement,which is time-consuming and high-cost.Computer vision technology can make up for the shortage of manual measurement and provide technical support for the realization of intelligent growth parameters detection of apples on trees.However,due to the complex natural environment in which apples grow,and that fruit grows from small to large and from green to red over the whole fruit growth stage,it is difficult to accurately and automatically recognize the fruit on the tree and detect the size information of the fruit.To solve these problems,on the basis of the research results at home and abroad,using the digital image processing technology,pattern recognition,machine learning,and some related agricultural information technology and methods,this research systematically studied the recognition of apples before fruit thinning,apple image segmentation method that independent of color features,apple contour detection and apple‘s horizontal diameter remote detection method.The main contents and conclusions are as follows:(1)An apple recognition method before fruit thinning based on region-based fully convolutional network(R-FCN)was proposed.It is difficult to recognize the apple target before the fruit thinning,because the apples are small at that time and the color of apples is very similar to the background.To solve the problem,after analyzing the frameworks and recognition results of ResNet-50 based R-FCN and ResNet-101 based R-FCN,a R-FCN based on ResNet-44 was designed to improve the recognition accuracy and simplify the network.The ResNet-44 based R-FCN consisted of ResNet-44 fully convolutional network,region proposal network(RPN)and region of interest(Ro I)sub-network.ResNet-44 fully convolutional network,the backbone network of the R-FCN,was used to extract the features of image.The features were then used by the RPN to generate Ro Is.After that,the features extracted by the ResNet-44 fully convolutional network and the Ro Is generated by RPN were used by Ro I sub-network to recognize and locate small apple targets.After data augmentation,23 591 images were randomly selected as train set and 4 739 images as validation set to train the network and optimize the parameters.After training,the simplified ResNet-44 based R-FCN was tested on the test set which composed of 332 images.The experimental results indicated that the method could recognize clustering apples,occluded apples,vague apples and apples with shadows,strong and weak illumination on the surface.In addition,apples divided into parts by branched or petiole could also be recognized effectively.The recognition recall rate and precision were 85.7% and 95.1%,respectively.The average recognition time was 0.187 s per image.The F1 of the proposed method was increased by 16.4%,0.7% and 0.7%,respectively,when compared to Faster R-CNN,ResNet-50 based R-FCN and ResNet-101 based R-FCN.The average running time of the proposed method was improved by 0.010 and 0.041 s compared with that of ResNet-50 based R-FCN and ResNet-101 based R-FCN,respectively.The proposed method could achieve the recognition of small green apple targets before fruits thinning which could not be realized by traditional methods.It could also be widely applied to the recognition of other small targets whose features are similar to background.(2)An apple image segmentation method that independent of color feature was proposed.From young to mature,the color of apple changes greatly from green to red.To achieve the segmentation of apples with different colors,and to improve the intelligence level of the image segmentation,firstly,saliency using natural statistics(SUN)visual attention model was used to extract the saliency map of the original image.After threshold segmentation the saliency map,the centroids of the obtained salient binary region were extracted as initial seed points.Simultaneously,Laplace operator was used to sharpen the original image.Then globalized probability of boundary-oriented watershed transform-ultrametric contour map(g Pb-OWT-UCM)and Otsu algorithms were applied to detect saliency contours of the sharpened images.With the built seed points and the extracted saliency contours,a region growing algorithm was performed to accurately segment apples.A total of 556 apple images captured in natural conditions were used to evaluate the effectiveness of the proposed method.An average segmentation error(SE),false positive rate(FPR),false negative rate(FNR)and overlap Index(OI)of 8.4%,0.8%,7.5% and 90.5%,respectively were achieved and the performance of the proposed method outperformed other six methods in comparison.The method can provide a more effective way to segment apples with green,red,and partially red colors without changing any features and parameters.(3)An apple edge detection network that fused convolutional features of the ResNet-50 was proposed(DAC-FEB).To detect the edge of apples in images rapidly and accurately,an improvement had been made on the basis of the ResNet-50.The proposed network fused the convolutional features of every block from stage 2 to stage 5 of the ResNet-50,and new convolutional features was added in the network to eliminate checkerboard artifacts caused by up-sampling.A total of 903 images with apples of different degrees of maturity were acquired,and 160 of them were selected as a test set.The other 703 images were augmented by horizontally flipping,and rotating original images and horizontally-flipped images to 90°,180° and 270°.After data augmentation,a total of 5944 images were obtained,which were then used for network training and parameter optimization.The experimental results showed that the F1 score of the DAC-FEB method was 53.1% on the test set,and the average run time was 0.075 s per image,which was better than the other seven edge detection methods.The DAC-FEB method can remove complex backgrounds effectively and detect apple edges accurately with a near real-time performance.The method can extract the edges of single apples,occluded apples,apples in cluster,immature green apples,mature red apples,and semi-mature partially red apples.(4)To solve the problem of obtaining the growth information of apples in natural scene,an apple‘s horizontal diameter detection method based on edge detection was proposed.Based on the designing of the apple image remote timing acquisition hardware system with CMOS network video camera and PCs as the core,the horizontal diameter measurement method of apples on tree was designed.After image pre-processing,the designed DAC-FEB network was used to extract the edges of apples in images.To fifilter out irrelevant apples in the images,a point on apple to be monitored was manually selected from the pre-processed image as seed point,and the region growing method was conducted on the extracted edge maps to segment out apple to be monitored.Apples to be monitored were considered as a circle in images before June 23,2018,and the diameter of the circle was taken as the apples‘ horizontal diameter.After June 23,2018,apple edges were fitted with ellipses,and the length of the line segment between the intersection points of the edge of apples to be monitored and the straight line where the long axis of fitting ellipse lied were used as the apples‘ horizontal diameter.Finally,the apples‘ horizontal diameter in the images was converted to the actual apples‘ horizontal diameter using calibration balls.The experimental results showed that the horizontal diameter of the apples could be detected by our system from the date after apple thinning(May 16,2018)to apple ripening(September 23,2018).The mean average absolute error of the apples‘ horizontal diameters detected by our system was 0.90 mm,and it decreased by 67.9% when compared with the circle fitting-based method(2.80 mm).Our system can monitor the growth of apples on the trees effectively and accurately.The proposed method provides a reference for monitoring the growth of other fruits during the growth period,and it can be used to optimize orchard management.