Design And Implementation Of Computation Node In Distributed Stream Computing Platform

Nowadays,with the continuous development of the Internet and the Internet of Things,more and more people and devices are accessing the network.In the network,there is a huge amount of data flowing at all times.These data streams hide a lot of useful information about defense,technology,and business.But many high-value information has strong timeliness and requires real-time analysis and extraction.Handling these massive amounts of real-time data requires a high-performance distributed streaming computing system that provides real-time analysis of the results from the system.The performance of the computing node affect the delay of message processing directly,so low latency and high throughput are the core metrics of the distributed stream computing system.Under this condition,the streaming computing scenario requires a low-latency,high-throughput,and flexible computing node,which is the difficulty of distributed stream computing.Specifically,the main innovations and work of this thesis are listed as follows:1.In the daily use of the distributed streaming computing platform independently developed by the laboratory,the high data throughput and low message processing delay of the computing nodes cannot be satisfied at the same time.In this thesis,a computing node framework suitable for streaming computing scenarios is proposed by analyzing the characteristics of streaming data,the way of sending and receiving messages of computing nodes and the way of processing messages under the real-time streaming computing scenarios.This framework optimizes the way of calculating node's sending and receiving data and improves the way of data processing,thus improving the system's throughput and reducing the delay of message processing.2.Data in the data stream contains the characteristics of spatio-temporal correlation.The thesis designs and implements a lightweight user-mode thread library of M:N.By opening a large number of lightweight threads,users can perform parallel computation of data stream so as to make full use of the spatio-temporal correlation of data to accelerate data mining tasks.Work strealing scheduling algorithm is used in the scheduling of lightweight user-state thread library,which can switch lightweight threads quickly in the context of user-state,so that the multi-core CPU can achieve task load balance between different cores,improve the utilization of computing resources and reduce the memory usage of computing nodes.3.Diversity of application scenarios for streaming computing platform.In the thesis,computing nodes are designed to support multiple streaming computing scenarios in the form of plug-ins,and plug-in interfaces supported by the platform are defined.Users only need to implement plug-in interfaces to facilitate the use of the platform,thus achieving the scalability of the streaming computing platform functions.4.Finally,the function and performance of the designed and implemented computing nodes are tested and the test results are analyzed in detail.The test results show that under the scenario of streaming computing,the framework of computing nodes designed and implemented in the thesis can meet the needs of streaming computing with high data throughput and low message processing delay.