Research And Implementation Of Real-time Computing Platform Based On Streaming Data

In recent years,as many fields have higher requirements for data timeliness,the big data processing system has shifted from offline batch processing to real-time stream processing,so as to process the data immediately after it is generated and generate value.The usual real-time data processing flow is to use the interface provided by the real-time computing framework to implement the logic code of data processing,and then upload the program package to the cluster terminal to manually execute the startup script.This processing flow has many disadvantages.First,the cost of programming is high,and the processing of real-time streaming data is cumbersome.Second,the deployment of the program involves a lot of repetitive work,and the status of real-time computing tasks cannot be accurately monitored,and problems cannot be detected in a timely manner.This paper adopts the current mainstream Flink as the real-time computing framework.Using Flink to process real-time data also has the above-mentioned problems.The process of program writing,deployment,and monitoring is complex and cumbersome.At the same time,Flink cannot adjust the cluster allocation according to the actual resource usage during task operation computing resources,you need to manually adjust the computing resource parameters and restart the task.Aiming at the limitations of Flink engine resource allocation scheduling strategy and real-time streaming data processing,this article mainly introduces the following three aspects.First of all,in view of the problems existing in Flink engine resource allocation and scheduling,this paper proposes a DRAS-Flink dynamic resource allocation strategy.The dynamic resource allocation strategy analyzes the resource usage of Flink tasks in the past period of time,and then calculates the computing resources expected by Flink according to the set Ratio reservation parameters,and finally uses the Resource Scheduler service to redistribute and schedule computing resources,so as to realize the Flink tasks in Dynamic adjustment of computing resources during operation.Secondly,for the cumbersome real-time stream data processing process,design and implement a real-time computing platform,and integrate the development,deployment,monitoring,operation and maintenance of Flink tasks on the real-time computing platform.In terms of platform system design,the computing engine module,task development module,and task monitoring module are designed and implemented to reduce the difficulty of developing Flink’s real-time computing program and simplify task deployment,monitoring,and operation and maintenance operations.Finally,an experimental task is designed on a real-time computing platform to verify the dynamic resource allocation strategy.The experimental results show that,in the case of insufficient or excess computing resources,the dynamic resource allocation strategy can expand or shrink resources during the running of Flink tasks,and the resource usage after reallocation remains in a healthy state,and the stability of tasks and Data processing efficiency is maintained at a high level.At the same time,the functions of the real-time computing platform are perfect,and each function meets the design requirements.The research and implementation of the real-time computing platform has significantly improved the efficiency of using Flink to process real-time data,and the stability of Flink tasks and the timeliness of data are both guaranteed.