Design And Implementation Of Feed Stream User Behavior Statistics System

With the continuous updating of Feed streaming products and the amount of users that have tens of millions,it can become an excellent platform for the company's business departments to promote their products,the small traffic experiments of the strategy department,and the optimization algorithm of the recommendation department.As more and more departments enter the Feed flow,and each department has its own log specification,the format of its log is becoming more and more complex,and it will also lead to log coverage of other departments.Because the data format is complex,Resulting in the analysis of log fields will be quite time-consuming。Often,some simple needs,due to the time-consuming analysis,lead to the delay of completion time,and the result of output is not easy to be verified.The maintenance of code is also very poor.When logs are not developed according to specifications,or log content is covered by other business lines,there is no uniform data monitoring platform to monitor,alarm,and feedback the situation to the relevant people for the first time.There is no unified report platform and query platform for data to be viewed every day.In the data,due to different lines of business are stored in a large table,but only one partition to distinguish,so the first task is to need to be stripped of their log,and then converting field,according to the analysis,ODS,DWD,DWS,ads layer to construct the data warehouse,the logs were scattered,store according to the subject,that is convenient for the subsequent development of the business,also extended the overall business flexibility,and reduce the coupling degree between each other.In terms of business development,we build a platform integrating data visualization,big data query and data monitoring.The development framework is js+html+springmvc+mybatis,a small amount of data is stored in mysql,and a large amount of data is stored in distributed column storage.At present,the establishment of data warehouse improves the efficiency of solving the demand and ensures the accuracy of data output.Configurable reports have also been put into use,which can easily complete the production of reports and reduce duplication of development.The data query page has the function of fast query and improves the work efficiency.Data monitoring page can monitor abnormal indicators and save manpower costs.