The Design And Implementation Of Game Data Platform Based On Hadoop

The Game Data Platform is responsible for extraction, calculation, statistics and presentation of the data belonging to Baidu’s Game division. Great accomplishment has been achieved by Baidu’s Game division since established. The need of data statistics and analysis is more and more urgent as the data grows continually. Decisions and strategy should be laid down based on registration, login and payment data since there’s a great value contained. On this basis, Game Data Platform is proposed. The purpose of Game Data Platform is to contribute a multiple dimensions big data analysis system based on dimensional model, which should be able to supply timely and accurate data analysis and satisfy various data requirement. On account of the complexity and quantity of data sources, the system presents an integrated solution based on Hadoop.The system consists of two subsystems, primary data system based on Hadoop and data platform system based on J2EE. Primary data system is responsible for data ETL (Extraction-Transformation-Loading) and data dimension calculation making use of Hadoop; Data platform is responsible for required data calculation, store and presentation. Characteristics features such as customized report, task scheduling and multi-level permission management are also designed and implemented. Data platform consists of six parts:today summary, customized report, market promotion, operation statistics, configuration management and frequent report configuration, each of which is subdivided into several sub modules. The paper presents requirements analysis, architecture design, detail design and implementation of Game Data Platform.The author has contributed to the primary data ETL module and customized report module, and independently designed and implemented today summary module, task scheduling module, user management module, role management module, permission management module, main game management module and game group management module. The system has already been put into use and is supplying various and plentiful data reporting features, which covers more than85%of the data requirement. High praise has been received since the data feedback efficiency is improved and human resource is released.