Design And Implementation Of Real Time Log Analysis And Storage System Based On Storm

At present,many Internet companies need to use real-time calculation of big data,including real-time statistics,real-time analysis recommendations,real-time risk management,real-time monitoring of public opinion and real-time personalized recommendation.These applications will use the data generated by the log real-time processing.According to the requirement of real-time processing of log,this paper designs and implements a real-time log analysis and storage system based on Storm.This paper introduces the significance of real-time log analysis and the related technologies involved in the development process,and design the framework and functional modules of the system.The main point of this paper is real-time log analysis back-end framework.In the framework,the distributed Flume component is used to monitor several log files.Log information is sent to Kafka’s Sink by Memory Channel.Using Kafka as a middle ware between collecting module and log processing module to achieve the decoupling of traffic peak clipping and service module.This paper uses Storm technology in the real-time processing module,and reads the data from Kafka by Spout and realizes the analysis and processing module by the Bolt nodes.Spout encapsulates the read data into Tuple data and sends it to the downstream Bolt nodes,the system processes the stream data according to the created Topology,and sends the processed data to the upstream nodes to ensure the correct processing of the data.Then Storm analyses the date in real-time and make the date be persisted,this system uses HBase as the persistent storage of data,HBase has the characteristics of high performance and easy expansion,and the structure of HBase can change dynamically.In this paper,the system of real-time log processing is implemented.It can read data from different data sources and realize real-time calculation,analysis,storage of log,and a distributed,stable and extensible system for real-time analysis and processing of mass logs is provided in this paper.