High-Throughput Three-Dimensional Reconstruction System For Cotton Canopy With Recovering Internal Occluded Structure Of Branches And Leaves

In the process of modern agricultural production,agricultural machinery is constantly developing in the direction of intelligence and wisdom.The field management and protection of crops is increasingly dependent on the acquisition of crop growth information.The more accurate and timelier the crop biomass information is obtained,the more effective the production management can control the growth environment of crops,thereby increasing the yield and improving the quality of agricultural products.However,the existing agricultural machinery and equipment for field management（including sprayers,fertilizer applicators,etc.）generally have the problem of weak crop perception,and it is impossible to realize high-throughput and high-precision real-time three-dimensional reconstruction of crop groups,which seriously restricts the development process of fine management of agricultural production.It is imperative to develop a high-throughput,high-precision,miniaturized,and airborne crop real-time reconstruction system.The crop canopy structure is complex,the branches and leaves are flexible,the leaf shape is irregular,and there are a large number of branches and leaves in the canopy.These are the main difficulties that make the high-throughput crop three-dimensional reconstruction difficult to achieve.The traditional three-dimensional reconstruction technology cannot achieve rapid and accurate reconstruction of the crop canopy.In view of the above difficulties,this paper takes the three-dimensional reconstruction of cotton plants as the research object,focuses on the reconstruction of the branch and leaf structure of canopy internal occluded through neural network method,and innovatively combines semantic segmentation,autoencoder and generative adversarial structure to study the general methods of crop canopy information collection,segmentation and internal occluded reconstruction.Finally,an efficient three-dimensional reconstruction system of crop canopy based on RGB-D sensor is designed,and high-throughput crop reconstruction experiments are carried out on the mobile platform developed by the project team.The main research work includes the following four aspects:（1）Artificial cotton plant construction and canopy data acquisition and processing.In order to facilitate the experiment,several artificial cotton plants were built according to the morphological rules of cotton canopy.RGB-D sensor Real Sense L515 was used to collect cotton canopy for RGB image and depth point cloud.The collected RGB images of cotton canopy were enhanced and manually labeled to make a PASCAL VOC format data set.（2）Three-dimensional reconstruction of cotton canopy leaf point cloud.This paper innovatively proposes a Cascade Leaf Segmentation and Completion Network（CLSCN）that can reconstruct occluded leaf images,and a Fragmental Leaf Point-cloud Reconstruction Algorithm（FLPRA）that combines instance segmentation netw ork,generative adversarial network and three-dimensional reconstruction algorithm.Reconstruct a complete three-dimensional model of cotton plants with internal and external leaves.The experimental results show that the cascade network proposed in this paper can output high-quality cotton leaf masks in the front stage.The Mean Intersection over Union（MIo U）of the model reaches 84.65%,and the accuracy of the occlusion leaves in the back stage can reach more than 94.36%.The Chamfer Distance（CD）between the reconstructed cotton leaf point cloud and the original complete cotton leaf point cloud is less than 3.7×10^-4.（3）Topological reconstruction of branches and leaves inside cotton plants.In this paper,a method for reconstructing the topology of branches and leaves inside cotton plants is designed.On the basis of completing the reconstruction of point cloud of cotton leaves,the complete topology of branches and leaves inside cotton plants is obtained by reconstructing the topology of main stems and branches of cotton plants respectively,so as to realize the three-dimensional reconstruction of branches and leaves inside cotton plants.Experiments show that when the reconstruction depth is less than 0.4 m,the overall reconstruction accuracy of cotton plants is more than 70.29%.（4）Evaluation of the effect of high-throughput cotton canopy three-dimensional reconstruction experiment.Based on the RGB-D sensor,a high-throughput three-dimensional reconstruction system of crop canopy was designed and tested on the mobile platform developed by the project team.The stability of the three-dimensional reconstruction system of cotton canopy was tested,and its three-dimensional reconstruction effect on seedling cotton plants was evaluated.The experiment shows that the overall reconstruction accuracy of cotton plants can reach 66.70%,and the reconstruction time is 20 s.The crop three-dimensional reconstruction method proposed in this paper to recover the internal occluded structure of the canopy can better realize the high-throughput three-dimensional reconstruction of the internal occluded structure of the crop,which is of great significance for the system to understand the crop growth status,yield evaluation and disease control.