Research And Implementation On Support Batch Computing And Streaming Computing Big Data System

Since big data contains vast amounts of information, it can help users to gain more comprehensive and deeper insight into business processes through analyzing and mining the vast amounts of information and extracting the value of information. Correspondingly it improves the enterprise’s decision-making power. For the research of computing model of big data, it mainly focuses on the study of single computing model at present. However, in the practical application, most applications use only one computing model of big data which is unable to meet all the needs of business processes. Therefore, studying and implementing a dual pattern computing architecture that supports both batch calculation and flow calculation is the inevitable requirement to achieve business developing. The research of this dissertation is mainly about study of system theory model, big data maturity model, dual mode computing architecture, algorithm and model research of architecture application and the design and implementation of securities analysis system. The main work of this paper is as following.1) The dissertation extends the big data maturity model. Through analyzing the existing big data maturity model, it extends the existing models from the perspective of the development of big data related technologies. Meanwhile the two-dimensional maturity model of big data is designed, which is combined with the actual needs of big data development. Also the modules’details of the given model are described.2) The dissertation designs a dual mode computing architecture for big data. It studies two computing models of big data, and summarizes the advantages and disadvantages of these two computing models. Through combining the two-dimensional maturity model of big data, a dual-mode computing architecture for big data is designed, and the working principle and flow of the given architecture are described. Based on the above analysis, a set of architecture application algorithms are designed and a business importance model is verified. The algorithms are dynamic load balancing algorithm, multi-source asynchronous data fusion algorithm, data fitting algorithm, and multi-granularity generation algorithm.3) The dissertation proposes a three-dimensional model of securities data. It selects the financial securities industry which is one of the important applications of big data as the application scenarios and validates the feasibility of the proposed architecture. Based on the existing big data theory and the characteristics of securities industry, a three-dimension model of securities data is put forward. The multi-granularities of both time dimension and space dimension in the given model are represented by quotient space theory.4) The dissertation designs and implements a securities big data analysis system. Based on the dual mode computing architecture of big data, it designs the securities big data analysis system and chooses the stock data as examples to verify the feasibility of the designed system.In the dissertation, the system performance is analyzed and verified using the stock exchange data of all A stocks in China as the testing objects. The testing results show that the proposed dual mode computing architecture can meet the needs of stock exchange very well. The average gain of the screened potential stocks is higher than that of the market, which proves that the data analysis system is feasible.