Design And Implementation Of K12 Online Education Data Warehouse Based On Big Data Platform

In recent years,with the development of the Internet,more and more data has been generated.In order to process and tap the value of these data,big data technology has been rapidly developed and applied.Hive,as an open source data application built on top of a big data foundation platform,has the characteristics of high scalability,high fault tolerance,and free mode,which can well meet the needs of enterprise data warehouses.Therefore,various industries have begun to vigorously build data warehouses based on big data platforms,and use them to collect and process data to tap their potential value.Based on the business needs of K12 online education as the background,this thesis based on a full study of the business needs of the enterprise,designed the overall system architecture of the K12 online education data warehouse based on the big data platform,and carried out a data flow framework for the data processing process.Design and technical selection.Build and deploy a Hadoop cluster by using big data ecological open source components,write Flume,Kafka and other programs to collect classroom buried logs,and use the Java programming language to log Etl to clean,transform,and desensitize and connect to Hive.The existing Mysql business database data was imported into Hive through Sqoop,and the shell programming language was used to develop a data consistency verification script to verify the correctness of the data entered into the warehouse.Combined with the specific business needs of K12 online education companies,dimensional modeling was adopted,and a star pattern was selected for the hierarchical design and development of several warehouses.Finally,the design and implementation of the data warehouse were completed,and the functions implemented by the data warehouse were displayed and analyzed.This thesis has completed the design and implementation of K12 online education data warehouse based on big data platform,provided standard reports and Kanban displays,met the business needs of various business units,supported multidimensional data analysis through model building,enhanced information processing capabilities,and provided data Digging laid the foundation.