Big Data Analytics And Decision-making Assistant Based On Enterprise Transaction Logs

With the sharp increase of enterprises logs data quantity in traditional industries,demands of mining the value of logs increase as well.In the aspect of logs collection,analytics,storage,traditional logs processing technologies are confronted with more and more difficulties.They can neither meet the requirements of flexibly colleting,fast reading or getting and various analyzing for massive logs,nor effectively assist enterprises to make marketing decision.In order to promote the market competitiveness,enterprises need the help of more advanced logs big data analytics technologies and methods to process the logs and assist enterprises in making marketing decisions.Specific to the situation and deficiencies of current logs analytics systems and based on the application scenario of big data analytics on large chain shopping centers transaction logs,from the perspective of different segments and models of logs processing,this paper researched on key technologies for transaction logs big data analytics,and compared the strengths and weaknesses between different technical solutions in this scenario.From the views of customer consumption categories,real-time trading volume of shopping malls,trading characteristics of specific date,algorithms and processes for enterprise market segmentation decision-making assistant were given out.An integrative distributed logs big data analytics system was designed and implemented.This system is combined with three modules.Logs collection module is based on Apache Flume,an open source logs service.Logs analysis module is implemented with a message queue technology called Kafka,a distributed computing framework called Spark and a database layer called Phoenix.Logs storage module is achieved by using HBase to save data and using Phoenix to accomplish interface conversion.It can meet the requirements that logs are collected flexibly and efficiently,analyzed in three different forms,including real-time analysis,off-line analysis,on-line analysis,and accessed with high speed along with uniform interface.This can assist enterprise in carrying out market segmentation decision-making from multiple aspects.Using experimental data,this paper gave out a test and compared the performance of logs processing under different data size between this system and others.And based on the real-time analysis module,off-line analysis module and on-line analysis module of the system,it finished the experiments of logs big data analytics and decision-making assistance.The correctness and effectiveness of the system,the algorithms and processes are verified,and the practical application value demonstrated.