Design And Implementation Of Enterprise Big Data Automatic Processing Platform Based On Microservice

Under the background of information digitization and intelligence,enterprise big data processing platform is applied in enterprise data management.Enterprises integrate data acquisition,analysis,cleaning and visualization through the big data processing platform,which contributes to comprehensive insight into data changes,business analysis,and data collaboration between departments to improve office efficiency.At present,the existing enterprise big data processing platform adopts the architecture of monomer architecture,which is simple to realize.Data governance is also the governance mechanism integrating data collection,analysis and data cleaning.With the continuous expansion of business,the amount of data collected is increasing,the data complexity is increasing,and the business side has higher and higher requirements for data support.As a result,it is difficult to expand the business functions of the platform,and the efficiency of data collection decreases with the increase of data volume.As a result,the existing platform can no longer meet the needs of users.An excellent enterprise big data processing platform should be able to face the everexpanding business,carry out efficient iterative development,and effectively carry out data acquisition,data analysis,data cleaning and data visualization in the face of data sources in the era of big data.Therefore,the reconstruction and optimization of enterprise big data processing platform has become an urgent problem to be solved.Combined with the actual business situation of the enterprise big data processing platform,this thesis proposes to build an automatic enterprise big data processing platform based on micro-services,platform automation through configuration,and reconstructs and optimizes and upgrades the enterprise big data processing platform.The main work of this thesis includes:(1)In the face of the business requirements of the enterprise big data automatic processing platform,this thesis carries out a detailed requirements analysis,and elaborates the business requirements required by the enterprise big data automatic processing platform combined with the flow chart.At the same time,a use case model is built,and the functional and non-functional requirements involved in the platform are carefully analyzed and evaluated,so as to better understand the business requirements of the enterprise big data automatic processing platform,and ensure that the functional and non-functional requirements of the platform meet the customer’s expectations and business requirements.(2)For the micro-service architecture design of enterprise big data automatic processing platform,this thesis uses DDD domain driven design method to carry out fine division and design of micro-services.The idea of microservices is integrated into the design of ETL mechanism.In the design of ETL mechanism,the data acquisition,analysis,cleaning and visualization services are separated,so as to realize the isolation and independence of each other.The function modules of data acquisition,analysis,cleaning and visualization can be independently developed and deployed,and can also be upgraded and maintained more effectively,thus improving the flexibility and scalability of the whole system.(3)In the field of data acquisition,in order to improve the collection efficiency and accuracy,this thesis designs a set of automatic process,using micro-services for data acquisition and automatic scheduling.At the same time,this thesis proposes a dynamic adaptive scheduling algorithm based on micro-service,and improves the DSOM algorithm on the basis of genetic algorithm,so that the data acquisition can be automatically allocated to the appropriate Docker node.At the same time,the data acquisition progress and the running state of micro-service can be monitored,and the data acquisition situation can be mastered in real time,so as to realize a more efficient and accurate data acquisition process.(4)In order to verify the reconstruction and optimization effect of the enterprise big data automatic processing platform based on micro-services,this thesis implements and tests the automatic processing platform based on micro-services to realize the micro-service functions involved in the platform.The micro-services are tested.According to the test and analysis results of the platform,it is verified that the enterprise big data automatic processing platform based on micro-services has good scalability,and the collection efficiency and data quality have been significantly improved.