Recognition And Localization For Tomatoes Under Open Enviroments Based On Binocular Stereo Vision

The accuracy and real time performance of recognition and localization methods for fruits and vegetables is the key to the success rate and efficiency of fruits and vegetables harvesting robots. Recognition and localization of fruits and vegetables is much influenced by the environment factors under open environments, so it is always a difficult task in the research field of harvesting robots.Tomatoes are used as samples in this study. Image segmentation methods for tomatoes, recognition methods for occluded tomatoes, recognition methods for clustered tomatoes and localization methods for tomatoes were detailed studied by using binocular stereo vision technology. Some problems in the recognition and localization of tomatoes were solved in this study. The developed recognition and localization algorithms offered help for improving the vision system of tomato harvesting robots. Main contents and results were listed as follows:(1) Segmentation performance was compared among three image segmentation methods for tomatoes based on color difference, normalized color difference and color component ratio. The method based on normalized color difference and the method based on color component ratio could realize the image segmentation for tomato images captured under different lighting conditions; the method based on normalized color difference was better under darker lighting conditions; the method based on double ratios of color components was better under front lighting conditions.(2) A piecewise thresholding segmentation method for tomatoes fusing a recognition method for light spots was presented. Based on the dependency between R component values of pixels in the region of a tomato and the illuminance on the surface of the tomato, this method used both advantages of two methods:the method based on normalized color difference and the method based on color component ratio. Furthermore, the recognition method for light spots was also used. Employing this method, tomato image segmentation could be realized under different lighting conditions. Test results from184images in which there were750tomatoes with light spots on their surface showed that the success rate of image segmentation was93.6%. which was much better than the one of the method based on normalized color difference, which was onlv36.3%. The average running time was71ms. This method had good segmentation performance to tomato images which were captured under front lighting conditions.(3) Two recognition methods for occluded tomatoes were studied:the method based on circle regression and the method based on circle Hough transformation. Firstly, the two methods were both based on curvature analysis, and then edge points with abnormal curvature values were discarded, after that, occluded tomatoes were recognized through the circle regression method and the circle Hough transformation method for the remaining edges, respectively. Test results from220images in which there were tomaoes occluded by leaves or branches showed that for tomatoes which were occluded by branches or leaves slightly, the recognition success rates of these two methods were90.8%and89.1%, respectively. For tomatoes occluded moderately, the recognition success rates of these two methods were90.8%and89.1%, respectively. For tomatoes occluded seriously, two methods both had bad recognition performance. Overall, for occluded tomatoes, the performance of the method based on circle Hough transformation was much better than that of the other. The running times for two methods were both about100ms.(4) Four recognition methods for clustered tomatoes were presented:the method based on mathematic morphology, the method based on circle regression, the method based on Hough transformation and the method based on binocular stereo vision. The method based on mathematic morphology was realized through the operation of conditional corrosion and circular expansion. The method based on circle regression and the method based on Hough transformation was similar to those used in the recognition of occluded tomatoes. In the method based on binocular stereo vision, clustered tomatoes were classified into two types:overlapping tomatoes and adhering tomatoes, based on the depth difference between the front tomato and the back one in a same clustered region employing an iterative Otsu method. Then, adhering tomatoes were recognized using the circle regression method. On the other hand, overlapping tomatoes were recognized using the circle regression method for the edges of the clustered region in color image which was segmented into several segments by the edges in depth map. Furthermore, the number of edge segments was same to the number of tomatoes in this clustered region. Test results from138pairs of stereo images of tomatoes occluded slightly showed that the recognition success rates of these four methods were60.9%,69.0%,71.1%,82.5%, respectively. However, for tomatoes occluded seriously, the recognition performance of these four methods were all not good enough. The running times of these four methods were3,85.138, and500ms. respectively. Test results showed that the recognition performance of the method based on binocular stereo vision was good to both two types of clustered tomatoes. Otherwise, the recognition performance of other three methods was good to adhering tomatoes, but was not good to overlapping tomatoes. Overall, the performance of the method based on binocular stereo vision was better than other three methods, especially for overlapping tomatoes.(5) Correction models for3D measurement error were presented. Three stereo matching methods were studied:the centroid-based stereo matching method, the area-based stereo matching method and the combination stereo matching method.3D localization errors were analyzed for the results produced based on these three stereo matching methods, the main factor to depth measurement errors was analyzed, and correction models were also presented. Furthermore, the influence to the accuracy of3D localization for tomatoes caused by occlusion was also discussed. Finally, the real time performance of these three stereo matching methods was tested, too. Combination stereo matching included two stages:rough stereo matching and precise stereo matching. Disparity acquired through centroid-based stereo matching at the rough matching stage was used as the center of the disparity range which was used in the area-based stereo matching at the precise matching stage. Then a dynamic disparity range was acquired of which the center moved with the shoot distance. By this way. not only the accuracy of the stereo matching was promised to be similar to that of the area-based stereo matching method, but also the time costs of stereo matching were also reduced. The time costs reduced to one third of that of area-based stereo matching method when the shoot distances were ranged from300to1000mm. Test results from1349pairs of stereo images showed that:1) for three stereo matching methods,x coordinate measurement errors were all smaller:the range of x coordinate measurement errors based on three stereo matching methods were [0,12.8],[1.3,13.8].[1.3.13.8] mm, respectively; but y coordinate and depth measurement errors were larger; y coordinate errors increased and depth errors decreased as the distances got larger.2) After the correction using binary piece wise linear regression models for y coordinate prediction, the ranges of y coordinate measurement errors based on three stereo matching methods were [-7,-0.8].[-6.1.1.4],[-6.1,1.8] mm. respectively3) Tomato size was the main factor to the depth measurement errors.4) After correction using binary linear regression models for depth prediction, the ranges of depth measurement errors based on three stereo matching methods were [-10.4,28.7],[-6,12.8],[-5.6,5.3] mm, respectively.5) x and y coordinate measurement results were more easily influenced by occlusion than depth measurement results.6) The ranges of running times of three stereo matching methods were [0.4,4],[24,288.8],[9.6,98.8] ms, respectively.(6) Finally, softwares of the recognition and localization methods for tomatoes under open environments were designed based on the C program language. The application flow chart of the recognition and localization methods for tomatoes and its dynamic link library produced from the C program of the recognition and localization methods was designed. Moreover, a VB program interface which was used to test the DLL was also designed. Harvesting experiments were executed both in laboratory and greenhouse after the DLL was employed by the vision system of a tomato harvesting robot. Among23times harvesting attempts in laboratory, the number of successful recognition was18, and the number of successful harvesting was11. Moreover, the numbers of successful recognition and successful harvesting were9and8in10times of harvesting attempts in greenhouse. The harvesting efficiency was about30s per tomato.The above work provided method and technology foundation for improving the adaption performance of the vision system of tomato harvesting robots under open environments.