Research On Agricultural Large Data Processing Method Based On Internet Of Things

The promotion of Internet of Things in Agricultural Information,the huge sensor and RFID nodes,can better monitor the information in the production environment,but they will increase the amount of data collected,the data will appear massive redundancy,give us follow-up data processing and analysis has brought some problems,seriously affecting the performance of data processing and analysis of quality.Such a huge scale of data,is the traditional data processing methods encountered problems.With the pace of large data age,there have been many large data computing frameworks,such as Hadoop,Storm,Spark,many companies choose them,and made a successful case,they have also been applied in the field of large agricultural data.According to the characteristics of the data and the use of the scene,select the appropriate processing tools,data processing must be considered factors.For highly redundant data,how to do the pretreatment,and how to do the processing and analysis of large data,how to make it robust and efficient,cannot be ignored details.According to the theory and practical experience,this paper analyzes the data flow and data characteristics of large agricultural data based on Internet of Things.According to the mechanism of Spark processing data analysis,from the point of view of reducing data redundancy and starting from the optimization of large table association,based on BloomFilter data middleware,Spark large table association optimization method is proposed.The main researchwork of this paper is as follows:(1)Based on the Internet of things agricultural data acquisition,the acquisition of data is often redundant,to the analysis of the following problems,in this paper,based on the advantages of BloomFilter,we propose an optimization method for filtering redundant data on the basis of BloomFilter.(2)Spark's large data computing framework for streaming data processing,can handle the Internet-based agricultural data flow and meet the real-time requirements,but an optimization method is proposed for the problem that the efficiency of the connection between the two tables is usually not high and the problem of data skew occurs.(3)The above optimization methods are applied to practical applications.A system model based on Spark and agricultural IOT is designed,which is mainly combined with the optimization methods of 1 and 2.