Research On The Performance And Conformance Of Storm-based Streaming Data Processing Application

In the context of Big Data era,Data-intensive applications(DIAs)are an important application form for large-scale data processing;Streaming Data Processing Applications(SDPA)use stream data processing technologies(e.g.,Storm)as a supporting framework,and have become the main application method of DIAs for stream data,which is widely used in the fields of Internet of Things,large e-commerce,and cloud computing.Based on the model-driven idea,the development lifecycle of stream data processing applications should include modeling,deployment,monitoring and optimization of the computational topology.However,the current situation of stream data processing application development is that there is no effective policy guidance for the allocation of computational resources(e.g.,Task task process,concurrent threads,etc.)in the deployment phase,and there is no effective means to guarantee the computational conformance(between application operation and computational model)in the monitoring phase of stream data processing applications.To address the shortcomings of the lack of effective resource allocation strategies for stream data processing applications and the problem of guaranteeing computational conformance,this thesis conducts the following research work:(1)A performance analysis method for tuple processing delay and queue backlog length of stream data processing application is proposed.Firstly,the Storm model is defined formally,and the execution semantics of the Storm model is given using color Petri nets,and the execution semantics also considers the fault-tolerance mechanism of Storm.Then,based on the execution definition of this model,the Storm model is implemented using the color Petri net tool under relevant assumptions,and subsequently the tuple processing latency and component queue backlog length of Storm are analyzed to guide the allocation of resources such as parallelism hint(threads)by the performance analysis method.(2)A runtime guarantee method for computational conformance of stream data processing application is proposed.Based on the performance analysis,the computational conformance definition of the Storm system is given based on the execution semantics of the Storm model.Subsequently,we propose a runtime guarantee method for computational conformance of stream data processing application,which provides real-time feedback on the error situation during Storm computation and locates the error location.(3)Design and implement a computational conformance monitoring platform for stream data processing application.The platform integrates the functions of topology modeling,deployment and computational conformance monitoring of stream data processing application,and encapsulates the process of model parsing and code compilation,realizing the cycle from modeling,deployment,monitoring and optimization of stream data processing application.The performance analysis method proposed in this thesis can guide the stream data processing application for effective resource allocation,and the computational conformance of the stream data processing application can be guaranteed by the computational conformance guarantee method,and the computational conformance monitoring platform can greatly improve the development efficiency of the stream data processing application.The experiments show that the average error of the tuple processing delay analysis method is controlled within 6% compared with the tuple processing delay of the actual stream data processing application.Through the queue backlog length analysis,the reasonable increase of component concurrency can improve the performance of stream data processing system.The runtime guarantee method of computation conformance for stream data processing application proposed in this thesis verifies the effectiveness and practicality of the method by analyzing and locating incorrect computations in event logs with a maximum throughput capacity of 21.32W/S through simulated experimental data and comparison with computation engines Esper and Drools.Also,the effectiveness and practicality of the stream data processing application computation conformance monitoring platform is verified by example.