| With the rapid development of China’s national economy,the financial leasing industry is also developing rapidly,and its comprehensive strength and competitiveness have gradually increased.It has become the third largest financing mode after capitalism and bank credit.In the era of big data with the rapid development of the Internet,the financial leasing business of financial leasing companies continues to expand,and customer information data and business data are increasing day by day.The traditional relational data warehouse adopts centralized architecture,and has the shortcomings of high cost and low efficiency in massive data storage and query analysis.The essence of financial leasing business is the same as bank credit,and financial leasing companies also face the credit risk of customer default.Therefore,how to solve the storage and management of massive data,and analyze and mine the value of business data,has become an urgent problem for enterprise managers.In response to the needs of financial leasing companies for massive data storage management,query analysis,and customer credit risk management and control,this paper has completed the implementation and application of the Spark-based financial leasing data warehouse.This paper integrates big data and data warehouse technology to extract and store the data of each system of financial leasing business,and uses distributed extract transform load(ETL)technology and dimension modeling method to design and implement the data warehouse of financial leasing enterprise on spark distributed platform;Combining data warehouse and data mining technology,this paper analyzes and mines the customer credit risk management data,constructs the customer credit risk evaluation model,and realizes the prediction of customer credit status.The main work of this paper is as follows:(1)The distributed ETL technology based on Spark is used to complete the ETL process of the internal data of the financial leasing company,and the original data that is discretely distributed in various business systems and channels is extracted and integrated and stored in HDFS and Hive tables.The extraction,conversion and loading of massive amounts of heterogeneous data,compared with the traditional distributed technology based on Map Reduce,the distributed ETL technology based on Spark improves the processing rate of massive data.(2)According to the development process of the financial leasing business and the data warehouse modeling method,three representative analysis themes were selected,and the corresponding dimension tables and fact tables were designed,and the design of the star model of the financial leasing data warehouse was completed.Designed and implemented a Spark-based financial leasing data warehouse.The bottom layer of the data warehouse is also a new generation of computing engine Spark.The distributed ETL method based on Spark is used to complete the data ETL process,and HDFS is used to store the massive data and intermediate results of the enterprise,and use Hive to define the tables of the financial leasing data warehouse,and use Spark SQL to perform query and analysis operations.(3)Combine data warehouse and data mining,use machine learning algorithms to analyze and mine customer credit data in the data warehouse,and propose a construction for the two problems of high-dimensional sparse customer credit characteristics and sample imbalance in the big data environment based on the credit risk assessment model of the RF-FL-Light GBM algorithm,it predicts the credit assessment of leased customers.Use Random Forest(RF)to sort and filter the importance of high-dimensional features,eliminate features that are likely to cause model overfitting and redundant uselessness,and determine the feature variables of the model;select and improve the currently widely used high-performance classification algorithm Light GBM uses the improved two-category balanced cross-straight loss function(FL)based on the Focal Loss function as the loss function of the Light GBM model to improve the situation that the imbalance of positive and negative samples leads to the reduction of model accuracy.Experiments with the model using historical customer data of the enterprise show that the credit risk assessment model based on the RF-FL-Light GBM algorithm greatly improves the classification accuracy of predicting default users,and its operation speed is faster.(4)Application demonstration and performance experiment of financial leasing data warehouse based on Spark.The financial leasing data warehouse analysis system is implemented,which mainly displays the effects of data warehouse applications in three aspects: interactive query,visualization and data mining;through data extraction and data conversion efficiency comparison experiments,it proves the performance of ETL technology based on Spark Efficiency: Through the efficiency comparison experiment of query analysis,it proves that the query performance of the financial leasing data warehouse based on Spark is higher and the practicability is stronger. |
