Study On Rice Yield Estimation Methods Based On Multispectral And Imaging Fusion Technology

Rice is the main food for about half of the world’s population and the second largest crop in China’s grain cultivation.It plays an important role in Chinese food security strategy.Efficient and accurate estimation of rice yield is of great significance to the formulation of national grain policy,precision agriculture management in the production process,rice breeding and cultivation research,and crop phenotypic analysis research.The traditional field yield sampling survey method mainly relies on artificial destructive sampling,which is time-consuming and labor-intensive,and the results are easily subject to subjective influence;The agrometeorological yield estimation method requires long-term agrometeorological monitoring data,and the crop growth model yield estimation method also requires crop biological parameters.These two yield estimation methods based on agronomic mechanism have too many input parameters,and it is difficult to obtain data,resulting in unstable model estimation accuracy,poor adaptability,and high cost.In recent years,the rapid development of optical technology and optical sensors have provided new solutions for crop yield estimation,and the rise of artificial intelligence algorithms has laid the foundation for rice yield estimation.By taking rices with multiple varieties and fertilizer treatments as the research objects,this paper aimed at building the efficient and accurate rice yield estimation system based on spectral imaging technology,RGB imaging technology,image processing method and deep learning method from the multi-scale of rice grain,panicle,and canopy.A dual light imaging system with thermal infrared and visible light was constructed,and the image registration algorithms,the thermal infrared de-ghost and multi-level nested conglutinated segmentation algorithms were proposed to achieve efficient and accurate detection of rice seed setting rate;Three strategies of transfer learning network were designed to establish a field rice panicledetection and counting model.Based on the detection models of rice seed setting rate and rice panicle count,the grain-panicle-field multiscale rice yield estimation framework was constructed.Multiple spectral indices and texture indices closely related to rice yield were newly constructed,and the yield index YI_Fusion was proposed by fusing multiple indices;A decision strategy for rice canopy RGB image classification modeling and normalized weight was proposed,and a rice yield estimation framework based on UAV image perception was constructed.The main research contents and results are as follows:（1）A dual light imaging system was established,and the image registration algorithm,thermal infrared artifact removal algorithm,and multi-layer nested grain adhesion segmentation algorithm were developed to achieve the detection of rice seed setting rate.Due to the small difference between the appearance characteristics of full and empty grains,and the difficulty of recognition through the image processing method based on manually selected color and texture characteristics,a dual-light imaging system with image acquisition and data storage functions was constructed by selecting thermal infrared camera and RGB camera to collect thermal infrared images and RGB images after the rice seeds were heated,and the temperature change laws of the two kinds of seeds after heating were studied.When the water content of grains is 11%-20%,the specific heat capacity is 4.9-16.3kj·kg^-1·K^-1.Because the weight of the full grain is larger than that of the empty grain,the full grain contains more heat.Through analysis,it was found that when the temperature of grain dropped from 35℃to 22℃（room temperature）,the temperature of empty grain was significantly lower than that of full grain,and the temperature difference between them first increased and then decreased,reaching a maximum of 2.8℃at 30s,and exceeding 1.7℃at 60s.The image registration method,thermal infrared de-ghost method and multi-layer nested adhesive segmentation algorithm were proposed to solve the problem of rice seed setting rate detection,and compared with three deep learning methods to verify the seed setting rate detection performance.The results showed that compared with the image registration method（86.78%）and the deep learning YOLO V3 method（96.75%）,our rice seed setting rate detection method based on the fusion of thermal infrared de-ghost and multi-level nested conglutinated segmentation algorithms had the best effect,and the seed setting rate detection accuracy reached 97.66%.（2）A deep learning network,Panicle Net,was designed to transfer the feature knowledge learned from the wheat panicle source domain dataset to the rice panicle target domain based on three migration learning strategies to construct a rice panicle detection and counting model,achieving rapid rice panicle detection and counting in the rice field.Aiming at the problem of poor anti-interference ability and stability of panicle detection algorithms based on manually selected features,and the sharp increase in costs caused by massive data annotation based on deep learning methods,the source domain dataset and the target domain dataset for transfer learning were constructed using the wheat panicle image public dataset and and the rice panicle image data collected through experiments,respectively.The transfer learning network,Panicle Net,was designed based on Res Net50 using the residual idea.Three transfer learning strategies,Panicle Net-F based on fine-tuning strategy,Panicle Net-C based on deep correlation alignment strategy,and PanicleNet-D based on deep domain confusion strategy,were proposed.The feasibility of transferring the panicle feature representation knowledge learned from the source domain dataset Wheat Dataset to the target domain dataset Panicle Dataset for rice panicle detection was studied.In order to solve the problem of repeated counting of rice panicle caused by image segmentation,a fusion algorithm was designed.The analysis of experimental data results showed that the precision,recall,and F1 scores of PanicleNet-F rice panicle detection were80.13%,75.64%,and 0.53,respectively,significantly superior to PanicleNet without fine tuning.The precision and recall of the fine tuning model trained based on small data in a 10%target domain dateset had a relatively small decrease,which proved that the fine tuning strategy could transfer the feature knowledge learned from wheat panicle detection to rice panicle detection.Panicle Net-D rice panicle detection precision,recall rate,and F1 score were 88.59%,80.37%,and0.84,respectively.Compared to PanicleNet-F and PanicleNet-D,Panicle Net-C had the best detection effect,with rice panicle detection precision,recall rate,and F1 score of 89.53%,92.85%,and 0.91,respectively.On the basis of completing the digital detection of seed setting rate and rice panicle,the thermal infrared de-ghost algorithm,multi-level nested conglutinated segmentation algorithm and the field rice panicle recognition and detection transfer learning network PanicleNet-C studied in the previous chapters were integrated to complete the parameter detection of seeds and rice panicles,and to build the grain-rice panicle-canopy type rice yield estimation framework based on unmanned aerial vehicle platform.The results showed that the average absolute percentage error（MAPE）of the rice yield estimation framework was 17.12%.（3）Combining the newly constructed three types of indices,the rice yield index YI_Fusion was developed to achieve rice yield estimation based on unmanned aerial vehicle multispectral images.Aiming at the problem that the accuracy of yield estimation models was often affected by the differences of rice varieties,fertilizer treatment methods and environmental conditions,resulting in low robustness,six new vegetation indices（VI）and six new color indices（CI）were proposed based on 5-band multispectral images,and six new texture indices（TI）were constructed using the gray level co-occurrence matrix of multispectral images.The correlation coefficients between three types of new indexes and rice yield was studied,and the indices with the highest correlation ranking were selected as modeling factors.The results showed that the performance of the newly constructed VI,CI and TI were better than that of the original vegetation index and single texture feature calculated at specific bands.The best band combinations of six VIs at heading stage are VI_{1（G-Red,B-G）}（R=0.5032）,VI_{2（Red,R-RE）}（R=0.6390）,VI_3（Red,G）（R=0.6282）,VI_4（Red,B）（R=0.7036）,VI_{5（RE,Red）}（R=0.4602）and VI_6（NIR,B）（R=0.4903）,The best combinations of six VI bands at maturation stage are VI_{1（RE-NIR,B-Red）}（R=0.7742）,VI_2（Red,B）（R=0.5921）,VI_{3（NIR,Red）}（R=0.6741）,IV_4（NIR,B）（R=0.6432）,VI_5（Red,B）（R=0.8423）,VI_{6（NIR,Red）}（R=0.6402）.The CI that have the highest correlation with yield at heading stage,maturity stage and all stage are CI₂,CI₃,and CI₁,respectively.The TI that have the highest correlation with yield during these three stages are TI(CON_NIR,COR_Red),TI(CON_NIR,CON_Red),and TI(CONT_NIR,COR_Red),respectively.The performance of VI-based yield index（YI）built by quadratic nonlinear regression model（QNR）in maturity stage is better than that of other stages,and better than that of CI-based and TI-based YI.The stepwise fusion strategy of modeling factors was designed to explore the performance difference between the multiple linear regression method（MLR）and the random forest method（RF）to construct the yield index YI_FUSION.The experimental results showed that,compared with the heading stage and the all stage,the yield index,YI_FUSION,was constructed by integrating 6 VIs,6 CIs and 6 TIs through the random forest algorithm at the maturity stage,with the best performance（R²=0.84,MAE=714.55 kg/ha,MAPE=7.86%）.It was proved that the feasibility of the image-spectrum fusion strategy based on UAV in yield estimation.（4）Based on a deep learning algorithm,a decision strategy for RGB image grading of rice canopy and normalizing the weight in the rice field was proposed,and the framework for estimating rice yield using UAV image sensing was constructed.Aiming at the problems of excessive parameters,low efficiency,and unstable accuracy in the rice yield estimation process,how to fully excavate rice image information,break through the traditionally fixed modeling idea of regression fitting,and propose new modeling ideas and strategies is the key issue to improve the accuracy of rice yield estimation.Using 10%of the average yield of the experimental rice plot as the grading interval,the sub plot images were graded based on the measured yield label data to form a data set for yield grading detection.Based on the deep learning network ConvNeXt,the yield estimation method based on unmanned aerial vehicle（UAV）rice canopy image classification was proposed for the first time.The image classification accuracy of the model in the test set was 90.17%.The results showed that the yield estimation performance based on ConvNeXt image classification model（MAPE=3.96%）was better than the regression model（MAPE=6.75%）.It also shows that the method of manually setting an interval of 10%of the average yield to grade the yield of the plot image can effectively control the yield estimation error.In order to further improve the estimation accuracy and make full use of the confidence scores of the yield classification model,the normalized weight decision-making strategy was introduced in the study,and a k-weight strategy was proposed through descending the confidence scores and weight normalization processing.The effects of different weight strategy choices in the range of2-weight to 5-weight on yield estimation performance were studied.It was found that the proposed normalized weight decision-making strategy had significant correction effects on yield estimation of incorrectly classified rice plot images,and could effectively reduce MAPE to 3.79%.Based on the field rice canopy RGB image classification and the decision-making strategy of normalized weight,the rice yield estimation framework based on UAV image perception was constructed.Through model generalization research,it was proved that the rice yield estimation framework can still maintain good generalization in different regions and different varieties of data,and the MAPE can reach 4.54%.