The Design And Implementation Of The Mass Sensor Data Acquisition System For Smart Agriculture

Smart agriculture requires large-scale sensors to acquire data such as air temperature,air humidity,soil moisture,soil nitrogen-phosphorus-potassium,carbon,dioxide and so on.But these data formats are different from each other,which resulting in high cost of development.The number of sensors reaches more than 100 thousand,which bringing great pressure of transmission and reducing the data reliability.In addition,a large amount of sensor data processing needs high concurrent data storage service.Based on the aboves,this paper,in a major national special project,designed and implemented a mass sensors data acquisition system of smart agriculture,MSDAS for short,referencing sensor data storage,Linux kernel network interface and an asynchronous network communication framework called Netty.This system has three layers,they are data storage,MSDAS gateway and sensors.It supports massive data fusion processing,unified management and maintenance,multi-party participation,and includes sensor data analysis based on metadata of sensor,MSDAS gateway communication network,sensor data storage services and other core abilities and components.The last,this paper has taken a thousands acres farmland as test area,thousand sensors and hundred MSDAS gateway deployed and simulated hundred thousand sensors and thousand MSDAS gateway to perform capacity of this system.The test and application show that the MSDAS system is feasible and effective.The main points of this paper are as follows:(1)Metadata based sensor data parsing.Sensor data acquisition for smart agriculture needs to support data heterogeneity of different types,models and manufacturers of sensors using different data encapsulation methods.This paper proposes and implements a metadata based sensor data parsing scheme.By developing metadata describing the sensor data format,MSDAS gateway parses user submitted sensor metadata and generates the corresponding type of sensor data parser.The implementation indicates that MSDAS now supports ten types and dozens of models in each type sensors,and the expansion of freedom.(2)Data transmission based on MSDAS gateway communication network.Sensor data acquisition for smart agriculture needs to support hundreds of thousands of sensors scale data transmission pressures.This paper presents a design and implementation of a forest like communication network consists of MSDAS gateway.MSDAS gateway can be real-time monitoring the rate of data transmission and change the subsequent routing,in order to avoid hot spots of MSDAS gateway data congestion.MSDAS gateway can also filter the data in the TCP/IP layer,in order to reduce the amount of data to be transferred,optimize the transmission efficiency.The application shows that the MSDAS system can support hundreds of millions of networks data transmission requests per day.(3)Mass sensor data storage based on MongoDB.Tens of thousands of scale sensors will produce hundreds of millions of sensors data everyday.The storage of these massive data needs to support high concurrency,high scalability and high availability.This paper presents a design and implementation of a massive data storage service based on MongoDB.The storage service uses a hard load machine to dispatch sensor data to the host with polling method,uses Netty asynchronous communication framework to receive and preprocess data,uses MongoDB Sharding method to build database.The application shows that MSDAS can support about 200 million data storage requests per day.