The Key Technology Research Of The Distributed Stream Architecture DSAR

In recent years,the Internet emerging technologies including big data,cloud computing,Internet of things,mobile Internet,have become the focus of widespread concern in the international society.It has attracted people's attention in the financial,military,scientific research,communication,medical and other fields.Nowadays the current mainstream runtime environments of big data and cloud computing are deployed in the Internet data centers(IDC)to provide the unified service for users.Providing computing services based on IDC has become the important technical means.However,the current mainstream data centers are dominated by the high cost,the limited resources and the huge power consumption,which faces the challenge of the generally rising demand of the future society for ubiquitous computing.Meanwhile,there are lots of heterogeneous and dynamic resources available distributed widely around people.Therefore,leveraging resources of the heterogeneous and dynamic network to build the efficient,reliable and low-cost distributed computing environment would be considered.It is significant for building the ubiquitous computing network environment in the future society.With the rapid development of information society,to provide the distributed computing service on the heterogeneous and dynamic network,people have to face at least three fundamental challenges of network resources: heterogeneity,dynamics,the long communication delay and the limited communication bandwidth.However,cloud computing,grid computing,P2 P computing,and other influential mainstream distributed computing models still have certain deficiencies for these challenges now.As the stream computing architecture has widely spread in recent years,this thesis deeply analyzes the characteristics of the stream computing model and considers it has features of the pervasiveness of computing resources,the lazy computing resource binding,the high data parallelism,and the pipeline parallelism.It has the significant potential for the challenges of the heterogeneous and dynamic network.This thesis firstly proposed a novel distributed stream architecture based on the traditional stream computing model.Oriented for the distributed computing in the heterogeneous and dynamic network,this thesis mainly carried out the following four aspects of research: the computing model,the programming model,the resource management technologies and the performance optimization techniques,the fault-tolerant techniques.The main work and the innovation points are as follows:1.Proposing the distributed stream architecture(Chapter 2)This thesis deeply analyzes that the traditional stream computing model has features of the pervasiveness of computing resources,the lazy computing resource binding,the high data parallelism and the pipeline parallelism.It is considered as a potential technology to solve the distributed computing model in the heterogeneous and dynamic network.Therefore,this thesis firstly proposes a novel distributed stream architecture,extending the stream computing model in the distributed environment.Then we propose the fundamental concepts of the distributed stream architecture,and design the new software stack of the distributed stream architecture to support the distributed stream processing.Meanwhile,this thesis builds the distributed computing model based on Kahn process network,and uses the CCS process algebra to analyze the semantic of this computing model.Based on the distributed stream architecture,the researchers can easily build the efficient,reliable and low-cost environment for the distributed computing in the heterogeneous and dynamic network.2.Proposing the distributed stream programming model(Chapter 3)Due to the shortages of the traditional stream programming languages,this thesis proposes a novel distributed stream programming model Brook# in the distributed stream architecture.It can solve the programming model for the heterogeneous and dynamic network.Then the thesis makes the comprehensive analysis of the Brook# translation technologies,mainly including the Brook# compiler and its runtime,such as the intermediate list method and the Brook# compilation directives.We focus on four Kernel execution modes and their implementation mechanism.The researchers can use Brook# to write efficient programs for the DSAR applications,which can help programmers to pay more attention on developing the performance,especially multiple level parallelism,regardless of details of the underlying resource heterogeneity,the dynamic task binding and the dynamic load balance.The experimental result shows that the distributed stream programming model Brook# can efficiently improve the execution performance of the applications,fully exploiting the potential parallelism of the applications in the distributed environment.3.Proposing the resource management technology,the performance model and the performance optimization techniques of the distributed stream architecture(Chapter 4)To solve the resource management of numerous resources on the heterogeneous and dynamic networks,the thesis firstly develops the network resource management system of the distributed stream architecture based on the master-slave framework and the two-level resource scheduling model.Then,this thesis analyzes the performance models of the four Kernel execution modes and the distributed stream architecture with the performance evaluation formulas presented.They also indicate the performance bottleneck and the performance optimization principles.Then,this thesis proposes two optimization techniques: the stream Eager transmission technique and the speculative Kernel execution technique.The experimental result shows the effectiveness of the resource management technology and the two optimization techniques,which can really effectively improve the performance of applications and reduce the execution cost of the programs.4.Proposing the fault-tolerant technique of the distributed stream architecture(Chapter 5)With the widely existing of the commercial devices,the continuous expansion of the computing system scale and the quickly emerging of the heterogeneous systems,the reliability of the distributed stream architecture is facing some serious challenges.This thesis analyzes the node fault model and common error type of the distributed stream architecture,especially the imperceptible soft error induced by the fault of the nodes running the stream-level program.Then,this thesis considers the feature of programs in the distributed stream architecture and introduces the fault-tolerant program section to efficiently control the extent of the error propagation and flexibly specify the fault-tolerant granularity.We propose the triple modular redundancy fault-tolerant technique(TREFT),which uses three same program copies for the error detection and correction efficiently on the available resources.The experimental result shows the flexible fault-tolerant technique can effectively reduce the fault-tolerant cost and improve the efficiency on the heterogeneous and dynamic network.