Design And Implementation Of E-Commerce User Profile System Based On Streaming Batch Integrated Architecture

Efficient and rapid construction of user profile is conducive to e-commerce business to achieve precision marketing and personalized real-time services.Offline batch data and realtime stream data together form the data base of user profile,which corresponds to the stable static attributes and changing behavior information of users.Therefore,the user profile system requires two data processing capabilities: stream and batch,stream-batch integration refers to the integration of two different data processing methods,stream processing and batch processing,to provide unified support for mixed bounded and unbounded data streams.This thesis focuses on the stream-batch integration data processing scheme and its implementation in the process of user profile construction under the e-commerce business scenario,applies the stream-batch integration data processing to generate user profile features,designs the algorithm model based on feature combination to realize the label prediction of user profile,and completes the construction of user portrait system.The main work of this thesis is as follows:(1)Demand Analysis and Outline Design of Stream-batch Integration User Profile System:the demand analysis of user profile system is carried out from both functional and nonfunctional aspects by integrating the characteristics of multi-source and massive original data of user profile.Then,according to the definition and analysis of stream data and batch data,formal stream-batch integration computing mode.Focusing on the stream-batch integration computing model,design the stream-batch integration processing architecture and hierarchical storage of system modules,and define user profile data.(2)User Profile Data Stream-batch Integration Processing Scheme: the core of user profile data processing is the calculation and storage of batch data and stream data,analyzing the data processing process of the user profile system,and designing and implementing a user profile stream-batch integration data processing scheme from the aspects of calculation and storage.Aiming at the accuracy and data consistency of the existing calculation schemes,a stream-batch integration calculation scheme is designed and realized by unifying the calculation engine and ensuring the end-to-end data consistency;Aiming at the problem of repeated storage and conversion of existing storage schemes,a storage scheme integrating stream and batch is designed and realized by layering data logic and zoning synchronization based on unified metadata.The experiment results prove the effectiveness of the stream-batch integration scheme.(3)Label Prediction Scheme Based on User Profile Features: user profile label prediction includes feature generation and prediction model,around the user profile label prediction process,the user profile label prediction is completed.Aiming at the real-time and integrity problems of the existing feature generation scheme,based on the integrated data processing of stream and batch,a real-time splicing of feature streams and offline real-time feature association fusion scheme are designed and implemented.The test results prove the effectiveness of the feature generation scheme.The noise problem introduced by the algorithm model,the label prediction model DeepBiFFM is designed based on the idea of feature combination,and the potential correlation between features is fully exploited.Experiments show that the model achieves better performance than the benchmark model on commercial datasets.(4)Implementation and Test of Streaming-batch Integration E-commerce User Profile System: combined with the relevant schemes of stream-batch integration data processing and label prediction,and based on the relevant business of e-commerce,the e-commerce user profile system is constructed by using the front and back-end related development framework.Based on the system,the functional test and performance test of the system are carried out.The test results show that the prototype system has the characteristics of real-time,meets the actual business needs.