The Design And Implementation Of Log Analysis Based On Storm

As big data computing an important part of the massive real-time calculation, real-time analysis of log data is becoming an important demand for companies and products of Internet. The massive log data real-time calculation is used in real-time statistics, real-time recommendation, real-time wind control, real-time monitoring and personalized recommendation services and so on in the scene. According to the user the massive log data real-time processing requirements, the design and realize the one based on the Storm real-time log analysis system in this paper.The main work of this paper includes:1) The paper introduces detailed the research background and related to the technology of the system. At the same time, a detailed description of the design, function of the system, the overall system architecture and module architecture be described by the paper and focuses on the detailed implementation of the system before and after the end of the system.2) Distributed collection module uses Flume technology to realize multi-source heterogeneous data collection, distribution and storage. This technology has the advantages of distributed, scalable, fault tolerance and high performance.3) Asynchronous communication module uses Kafka technology to achieve the decoupling between the collection module and the processing module. The biggest advantage of this technology is high performance and large throughput.4) Real time data analysis module uses the Storm technology, Storm in the form of data flow processing, high performance real-time processing of a steady stream of data.The system solves the distributed collection of log data, real-time computing, complex data processing and result display problem analysis, to achieve the capacity of heterologous log data collection function, data asynchronous communication function, log data function of real-time processing and real-time query function. This system mainly includes the following function modules:distributed massive log collection module, asynchronous communication message module, real-time log analysis calculation module, storage module, the results show module and other core module, to provide a stable, scalable, distributed, high performance mass of log data real-time analysis system.