The Optimization Of Topology And Real-time Scheduling Strategy On Storm

With the rapid development of cloud computing,Internet of Things,mobile Internet,social media,Internet financial and other new information technologies and application mode,people have an increasing demanding for real-time data processing.The real-time streaming system of open source develops fast because of its high Storm reliability and good processing mode,which is widely used in many enterprise platforms.However,in Storm,there are several problems in urgent need to be solved.(1)Currently,there is no mechanism in Storm system to check Topology automatically,nor to adjust the Topology adaptively.(2)The default scheduling algorithm in Storm is pseudo-random round-robin scheduling algorithm.Without taking the specificity of the Topology into consideration,there are quite of load balancing issues in scheduling process.(3)The existing improved scheduling algorithm improves some part of the scheduling process,solving problems such as traffic problems,resource availability and Qo S problem,absent in solving machine load imbalance problem.It is of great theoretical value and practical significance to research Topology optimization strategy and improvements of tasks scheduling algorithm,on the base of the Storm real-time streaming system.For the above problems,it studies the following three aspects in this thesis:(1)In response to existing mandate Topology optimization method that can not implement adaptive adjustment,this thesis analyzes the existing methods of Topology optimization at the point of topological structure and topological parameters,and ultimately proposes the dynamic optimization algorithms of Storm Topology based on Theory of Constraints,which is combined with Ideological work processing pipeline Theory of Constraints.It is by modifying the task number parameters of Bolt assembly,receiving length parameter and transmit parameters of buffer-queue,removing performance bottlenecks to get the quest Topology performance bottleneck components and dig out other tasks performance bottlenecks that exists in the Topology.Wherein modifying the parameters of the process is dynamic,so that there is no need to stop the Topology.(2)In response to the existing scheduling algorithms that can not effectively solve the inter-node machine load balancing,this thesis analyses the scheduling process of default scheduling algorithm,and focus on the topology-based and traffic-based scheduling algorithm.Combined with load balancing allocation strategy,thread scheduling algorithm is proposed on the base of Topology and traffic.The scheduling algorithm is divided into two steps.One is,it divides the thread of various components in topological graph in accordance with Topology and inter communication between components.According to load balancing between the cluster nodes,assign slots to the machine node port.In this way,the algorithm can reduce the delay of the entire system,while traffic and throughput of the system is guaranteed.(3)Confirm the above two proposed algorithms in the experimental part.In the first algorithm experiment,it is done with using the word-count Topology.With the comprehensive indicator of system average latency and throughput,it uses the controlling-variable method to determine the best value of weight parameter ?,?.After optimizing the Topology with best value,it verifies the overall performance superiority in system throughput and the average delay;In the second algorithm experiment,determining the weight ? in scheduling algorithm is the principal main.By comparing this scheduling algorithm with Storm default scheduling algorithm,topology-based scheduling algorithm with traffic-based scheduling algorithm,it does verify the overall performance superiority in system throughput and the average delay.This scheduling algorithm comparing the default scheduling algorithm scheduling algorithm average latency of Strom reduced about 46%,,the throughput performance of Storm improved by about 91%,inter-node traffic is reduced about 40%.