Research On Task Scheduling And Resource Dynamic Adjustment Algorithm In Storm Cluster Environment

Storm is a big data streaming computing framework deployed and run in distributed clusters.It has the advantages of active open source community and good compatibility.The real-time requirements of streaming big data computing are high,so task scheduling methods and dynamic resource adjustment in the face of sudden and unknown traffic changes have become key issues.The Storm framework adopts the task scheduling mechanism of fair round-robin,which is a simple task scheduling method without comprehensive consideration of resource utilization and task communication cost.Streaming big data input is often accompanied by traffic peaks or troughs.Faced with this situation,if the dynamic adjustment of cluster resources is not performed in time,it will be difficult for the job to run,resulting in an increase in response time,or the cluster nodes will be idle,resulting in resources waste problem.In response to the above problems,this paper conducts in-depth research on task scheduling and resource dynamic adjustment algorithms in the streaming big data Storm cluster environment.The main research work is as follows:1)A method for obtaining task scheduling scheme by combining Multiobjective Discrete Fitness Distance Ratio Particle Swarm Optimization(MDFDRPSO,Abbreviated as MDFP)and Euclidean Distance Load Balancing(EDLB)algorithm is proposed.First of all,in order to solve the task scheduling problem in the cluster environment of streaming task computing,the particle swarm algorithm is scene-based,that is,multiobjective discretization,and the meaning and objective function of particles are designed.Secondly,the idea of fitness distance ratio is used to propose learning from neighbors.At the same time,a time-varying inertia factor and learning factor are designed to balance the search ability,adjust the search process,and obtain a set of optimal solutions for task scheduling.The EDLB algorithm decides the final solution according to the three-dimensional Euclidean distance model of the resource type in the solution set.The result of this algorithm is the final task scheduling scheme,which considers both improving resource utilization and load balancing,so that tasks can be reasonably allocated to nodes for running.2)In order to adapt to the change of flow during operation,a pheromone-based FSarsa(0)algorithm(PFSarsa(0))is proposed to make the decision of dynamic resource adjustment.Firstly,a prediction module is constructed,and the input value of the algorithm is predicted in advance according to the collected monitoring information.Then,the details of membership,rules,and immediate rewards in the fuzzy reinforcement learning algorithm were designed,and a complete fuzzy logic controller was constructed.At the same time,the pheromone based on the ant colony algorithm is used to record the historical information of the state change of the learning process,which guides the learning rate of the error to the next step in the iteration of the fuzzy reinforcement learning algorithm designed in this paper.3)By building a cluster and designing experiments,the experimental results show that the algorithm in this paper can improve the performance of the Storm system when running streaming jobs by reasonably determining the task scheduling scheme and effectively and timely adjusting the resources dynamically,and verify its effectiveness.