Design And Implementation Of User Analysis-oriented Log Processing System For Logistics Public Service Platform

With the continuous development of China's economy,the logistics industry has also grown rapidly,and the daily page view of the logistics public service platform is growing rapidly.Massive access log data may seem disorganized,but it actually contains much information of user habits.Through the deep information mining of interaction process between users and website,the platform can provide better service,meet the deeper needs,and retain more users,thus the significance and practicability of this paper are the guarantee.Therefore,this paper has the practical significance of improving the efficiency and reducing costs for the platform.This paper designs and implements a user analysis-oriented log processing system for logistics public service platform by using the user access log,achieves a system range from the log collection,storage and display the result.This paper mainly explores the following points:(1)Studied the related technology.The concepts of access log,Flume-NG and Hadoop has been studied.It mainly includes the feature of access log,the structure of Flume and some technology related to the Hadoop.Due to the effect of Hadoop,this paper makes a deep exploration.(2)Analyzed the requirements of log processing system of Logistic Public Service Platform.Based on current situation this paper analyzed the existing problems in platform,such as inability to store and analyze mass data and low utilization rate of data.This paper solved the problem of corresponding to the problems by combining with the content of access log,then put forward corresponding demand indicators for the function of the system.(3)Completed the design and implementation of log processing system.It consists of four modules:acquisition and storage module,cleaning and filtering module,data mining module and result display module.And finally this paper provided the design scheme of each module.In the implementation part,the corresponding code and processing logic are given.Finally,a Hadoop-based system experimental environment is set up,and each module's features are tested successfully using the given implementation way.Also,the visualization results are analyzed on the public platform of logistics stand.By using distributed computing technology to store and analyze user access log files of platform,the insufficient analytical capacity of traditional technologies in the face of large amounts of data is solved.Its analysis results can help the platform to know the current business development,the popularity of each web pages visited by the users,and the geographical distribution of visits by users,in order to provide data support for management.