Design And Implementation Of E-commerce Real-time Analysis System Based On Flink

In recent years,with the development of information and communication technology,all walks of life on their system data collection,processing and accumulation of more and more data,data volume is explosive growth.In this context,big data has developed rapidly in various fields,and the world is stepping Into a new era of big data.With the increasing amount of data,the need of efficient data storage and real-time analysis and processing is also mentioned.If we want to quickly and effectively analyze meaningful data information from massive data,we need a set of data real-time statistical analysis system that fits the actual business requirements.As an Internet industry,the e-commerce industry is also facing the same challenge in today’s big data era.The current mainstream e-commerce data analysis is based on nonreal-time data,which often stores T+1 data,which cannot meet the demand for realtime data processing.Some enterprises adjust the schedule to half a day or even less.This method takes a long time and has a high throughput,which only meets the requirements of scenarios with low timeliness.In order to meet the needs of real-time data statistical analysis in actual business,this paper realizes real-time analysis of user behavior data and business transaction data based on Flink technology.This paper is divided Into four modules,which are real-time data acquisition,data modeling,real-time data statistical analysis and data Interface visualization.Data real-time acquisition module in the log data by Nignx real-time acquisition to Kafka,business data using Flink CDC real-time synchronization to Kafka message queue,fact data and dimension data are respectively stored in Kafka and Kudu database in the next layer,by Flink pull Kafka on the business data for real-time processing.The calculation results are written Into Kudu in real time;Based on the actual demand scenario,data modeling is carried out for the received data,and a wide table with reasonable and high reusability is designed.The final statistical analysis is transmitted to the front Sugar screen for display through the Interface.This paper is mainly applied to the real-time analysis of big data in e-commerce industry.This system meets the requirements of online real-time processing and analysis of real-time Streaming data by operation personnel,who can Intuitively see the real-time analysis of mall marketing data on the Sugar screen.This system can make the online App’s error log warning get immediate response through the error log analysis module.At the same time,the system also provides model training data for the recommendation algorithm department,and the Interface and wide surface data of the system can be used as model input,providing indispensable data Sources for the realtime recommendation of the mall.