| With the rapid development of sensor,communication,Internet of things and big data technology,human beings have entered the 5g era of Internet of things.China is a large agricultural country,also a large agricultural machinery manufacturing and use country.Different agricultural production modes,poor field environment,complex production links and various influencing factors make the data produced by agricultural machinery operation have the characteristics of diverse storage media,complex structure,high dimension and strong timeliness,which is easy to generate outstanding problems such as large cloud computing load,slow response speed and data abnormality.Therefore,this paper studies the data cleaning and data processing methods of large-scale load balancing cluster,establishes the spatiotemporal data model and data platform of agricultural machinery operation plot,and optimizes the retrieval technology of agricultural machinery operation data.,the main research contents of this paper include:1.Classify the data of agricultural machinery operation,and establish a four-dimensional spatial-temporal data model of agricultural machinery operation plot.The model focuses on the location information,meteorological information,attachment information and plot operation information of different time and dynamic environment and Combined with WebGIS technology to visualize relevant data.2.Research on the abnormal problems of agricultural machinery operation data in the process of transmission,and propose a data cleaning algorithm based on Flink.The algorithm determines the abnormal data by variance constraint,and estimates the original abnormal data by solving the quadratic equation of one variable by the minimum variation principle,and calculates the optimal estimation value by ARX model iteration.When the amount of data reaches 1 × 105,the accuracy of the algorithm tends to be stable,the accuracy rate P is 0.94,R value is between 0.9-1,F value is 0.94,RMS error is 2.82;the optimization results show that:when the proportion of abnormal data is 5%,M=5,?=0.1,W=1000,The results show that the system can meet the best repair effect with acceptable time complexity when it is 1000,the accuracy rate is 0.95,the R value is between 0.9-1,the F value is 0.95,the root mean square error is less than 1,and the response time is less than 1 s,which solves the data abnormal problem in the process of agricultural machinery operation data transmission on the server side to a certain extent.3.HBase database is used to store agricultural machinery operation data,and the problem of poor performance of HBase in retrieving agricultural machinery operation data under multiple conditions is studied.This paper proposes a two-level non primary key index method based on Solr.When the data volume reaches 5 × 107,the response time is less than I s,and the optimized performance is 3 times higher than that of native HBase;the data scale is 1 × 105?1 × 106 and 1 × 107,respectively When the reading and writing capacity is 1 × 105?1 × 106 and 1 × 107 pieces of data,the optimized insertion performance reduces by 13.3%,which solves the problem of slow response of data retrieval to a certain extent.4.A solution of data platform with four layers of architecture is designed,which are perception layer,network layer,middle layer and application layer.The test results show that the average response time is 110ms,the longest response time is 155ms,the fastest response time is 22ms,the error rate is 0.28%,and the throughput is 94.5MB/s. |
PDF Full Text Request