Research On Scheduling And Load Balancing Based On Topology Under Storm Cluster

In the era of big data,data contains a wealth of meaning and value.Real-time computation is another typical computational model relative to batch processing,which has been widely used in the fields of Health and Life Sciences,telecom operators,ecommerce and speculative market.As a real-time computing framework,Storm has many excellent features such as scalability,high error tolerance,low latency,facility of maintenance,and is favored in the industry.This thesis investigates the status quo of scheduling and load balancing of Storm-a distributed computing framework,and studies the optimal scheduling strategy based on Topology.Specifically,the following research work is completed:In view of the fact that the Storm real-time computing framework ignores the logical coupling among the tasks of each component in Topology which reduces the efficiency of the flow event processing,this thesis proposes a local priority scheduling policy based on Topology,with the method of process algebra abstract Topology and transform it equivalent into a plurality of parallel logic model execution subsystem,using the model to implement local maximum scheduling.Compared with the default scheduling strategy,the experimental results show that the proposed scheduling strategy can effectively reduce the network overhead caused by the flow event processing,and improve the system processing efficiency.According to the problem that the Storm real-time computing framework in load balancing can not adapt to the heterogeneous clusters,defined the load model based on CPU.Again,the load balancing strategy based on Topology is proposed,which is based on the improvement of the local priority scheduling policy,and the load constraints are added to the two models,which are scheduling model and resource allocation.The experimental results show that this strategy can ensure the load balancing of CPU resources on heterogeneous clusters under the premise of optimizing the efficiency of the local priority scheduling policy.The research shows that the Optimizing Storm scheduling strategy can improve the flow event processing efficiency of 10%-20%,and can adapt to heterogeneous cluster resource overhead tasks,improve system efficiency and stability.