Design And Implementation Of Data Quality Monitoring Platform For Internet Financial Enterprises

With the development of computer technology and its wider application in the financial industry,data has become an extremely important asset of financial enterprises.Many Internet financial enterprises use big data technology to carry out data mining and analysis,such as user portrait,precision marketing,anti-fraud,anti-money laundering,and credit rating.However,due to the long process of data collection and processing,complex process,large data scale and fast update speed,data loss,data duplication,data inconsistency and other data quality problems frequently occur,these data quality problems will affect the accuracy of data mining and analysis.Therefore,in the context of big data era,how to quickly identify and solve data quality problems is very important.Aiming at the data quality problems existing in the data assets of Internet enterprises,this paper investigated relevant theories of data quality at home and abroad and designed and implemented a data quality monitoring platform based on the actual business needs,focusing on the following work:(1)By investigating the research status of data quality,data quality dimension,data quality assessment and data quality monitoring platform,ten kinds of quality inspection rules of the data quality monitoring platform are standardized from the perspectives of completeness,accuracy,consistency,standardization and uniqueness;(2)A data quality assessment model based on combination weighting method is proposed,which solves the problem that the weight of quality inspection rules in the quality assessment model of traditional data quality monitoring platform relies too much on expert experience;(3)Analyze the requirements of the data quality monitoring platform based on the actual business,and explain the overall requirements,functional requirements and non-functional requirements;(4)On the basis of demand analysis,the general design of the data quality monitoring platform,including the overall architecture design,functional structure design and database design;(5)Detailed design and implementation of the quality monitoring module,task center module,alarm management module,problem handling module and basic information management module of the data quality monitoring platform;(6)Design test cases for each functional module of the data quality monitoring platform,and test the system according to the test cases.The data quality monitoring platform provides comprehensive data quality assessment ability,real-time alarm monitoring ability and process-based problem handling mechanism,which provides a basic guarantee for enterprises to identify and solve data quality problems.