Design And Implementation Of A Data Acquisition And Real Time Processing System Based On Kafka

In recent years,with the rapid development of the Internet industry and the improvement of residents' living standards,people have higher and higher requirements for the safety of life and the quality of the environment.More and more people choose to travel in non-motor vehicles.Everyone hopes to live in a civilized,orderly,safe and harmonious environment.With the rapid development of advanced technologies such as the Internet of things,cloud computing,distributed and big data analysis and processing,local government departments are vigorously promoting the construction of "Internet plus" and improving the efficiency of regulatory services.The topic of this paper comes from the project of "urban vehicle intelligent prevention and control system".This system collects the information of urban electric vehicles by issuing electronic license plates for each electric vehicle and installing sensing devices on the main roads of each district and county,solves the problems of monitoring,management and tracking of stolen electric vehicles,and provides unified management of urban electric vehicles for a city public security bureau.As more and more electric vehicle data are received in districts and counties,vehicle data receiving and real-time data processing become a difficult problem in current projects.This paper in view of the "urban traffic intelligent control system" project data in the receiving quantity is more and more big,lead to data processing of response delay,data loss,the problem such as high memory load,using the Kafka has high reliability,asynchronous communication,lateral extension ability,Netty reducing resource consumption,enhance the ability of resource utilization,Spark Streaming to quickly analyze data and real-time response ability and optimize the data acquisition and data processing module,to achieve a high reliability and low latency of data acquisition and real-time processing system.Based on the analysis of the existing project background and technology,the main work content is as follows:(1)System requirements analysis.Model the platform system,analyze the requirements of the data acquisition and real-time processing system,and summarize the functional and non-functional requirements of the system.(2)System design and implementation.Design the overall architecture of the data acquisition and real-time processing system in conjunction with system requirements.Firstly,the asynchronous communication framework Netty is used to realize the access and parsing of data records.Then,based on Spark Streaming,the status maintenance of the data collector,key object arming and real-time alarm are implemented.Finally,the storage of data records is completed by using Hbase and Redis.Cache of hotspot data.(3)System testing.Deploy the system test environment,complete the functional test of the system,and compare the differences between the original system and the system in the comparison of data access,data compression and processing time.After system testing,the system implements functions such as terminal device status detection,basic device update,and arming alarm.At the same time,the system data compression rate is 33% higher than the original system,the data storage time is less than 30 s,and the stable access of 1000 machines can be guaranteed,which meets the high concurrent access index and achieves the system performance proposed by the user.demand.The test results show that the system is superior to the original system in many dimensions such as data access,data compression and processing time.