Design And Implementation Of Streaming Data Processing System

With the deployment of the global network of China-independently built Beidou-3 global navigation satellite system,China is the third country to build the global navigation satellite system indicator and China has reached the world-class level.Thanks to the high performance of the Bei Dou system,it is widely used in various industries.Therefore,it is important to study and analyze the data management of satellite navigation time.This paper originates from the National Time Service Center’s project.During working in the center,the data of the advanced satellite navigation test system must be aggregated and managed.The data management system will eventually display them.At the same time,it also provides data services to the other systems and it can fulfill the task of monitoring the working data of other systems to realize the function of the real-time alarm.However,in the current environment,the pressure of project complexity is becoming more prominent with the traditional data management methods.For example,the data source is growing steadily and the business needs are also increasing rapidly.Therefore,it is a great challenge that we can solve the problems of prolonged data processing,slow response,and memory pressure of massive data in the field of satellite timing.What’s more,to solve the above difficulties,this paper designs and implements a data management architecture that supports multiple concurrencies.In this architecture,the Netty framework is used to ensure the reliability and high performance of data collection,the Flink computing engine is used to deal with the real-time problem of stream data processing,and the distributed storage component Pravega is used to solve the storage pressure problem of massive data.According to the project background and the characteristics of the current technology,the three parts of this paper are included.(1)The system requirements analysis was carried out for the GNSS stream data processing system,and the interaction process between the user and the system,the data transmission process between the system and the external system,the visual front-end design of the system,and the requirements of the system in actual working scenarios were described.According to the non-functional requirements,the application software structure and technical scheme of the system are engineered according to the distributed idea,and the design of the Web Socket-based system communication data interaction scheme between some services of the system is completed.(2)In addition,in view of the delays faced by data acquisition,the system server and decoding chain are developed based on the Netty framework to realize the data reception and analysis of multi-port custom protocols,and reduce the system resource utilization by optimizing the relevant parameters.Aiming at the real-time problem in the data processing stage,through the performance comparison of various anomaly detection methods,an anomaly detection scheme based on the IForest algorithm of the isolated forest is designed,and the Flink computing engine performs parallel processing on the anomaly detection part of the data processing,so as to compress the processing.time delay.In view of the storage pressure of a large amount of data,Pravega is used to realize the storage layer structure of the system,and the performance of various binary serialization protocols is tested.(3)Besides,implement each module of the system,deploy the system in the environment,write test cases for each function in the system,and perform performance testing to verify the functional and non-functional requirements of the system,where the test results indicate that the system All functions and performance indicators meet the expected target requirements.