D-Map/Reduce Dynamic Topology Management System And Protocols

With the development of information, data is increasing, the research about large dataprocessing get more and more attention in various fields and produce more profound impacton our lives. In recent years, the rapid development of electronic banking, networking, socialnetworking and other areas of real-time processing for large-scale streaming data puts forwardhigher requirements. However, Hadoop system is designed for static data processing with filesystem, which is not well adapted to the needs of real-time continuous processing forstreaming data. The conventional stream processing models in which the job is described as aDAG diagram are lack of scalability and parallelism for large-scale calculating.In this paper, we present an Online Flexible MapReduce (D-MapReduce) processing modeland some dynamic topology protocols for the current problem of large-scale data processing.This model is compatible with existing MapReduce and supports dynamic change and elasticadjustment of topology. Finally the model can satisfy the characteristics including timelinessand dynamic and burstiness of streaming data. Compared with the traditional model, themodel in this paper has following characteristics:(1) Adapt the diversity of topology ofstreaming job based on the description of job with DAG.(2) Improve the real-time property ofsystem by the use of memory.(3) Expand the system elastically and compatible withMapReduce of multiple parallel computing tasks.（4）Based on the OTAP Protocol, weimplement the flixible system, for the surges.Hierarchical definition for model makes the scale of system is much larger and thetopology management is more complex. We design a series of protocols for topologymanagement including online initialization protocol and online adjustment protocol as well asthe further improvement of system and integration of resources. We propose a concept calledoperation sharing and design a job sharing protocol.Based on the description of system model and the definition of communication protocolabout topology management, we realize a verification Platform for large-scale streaming data.The communacation complexity and the flexibility have been verified by the Platform.