Design And Implementation Of A Distributed Log Analysis System

With the development of computer technology, log analysis becomes increasingly important, and has been effectively applied in many areas. Analyzing the running log can effectively improve the efficiency of maintenance work; also, log analysis is one of the means of active defense in system security; analyzing business data log is helpful for diagnosing and solving business problems. The EBupt Information Technology Co., Ltd. dedicates itself to the field of intelligent network communications; there are plenty log analysis applications. With the continuous development of business, the amount of data grows fast in many business log analysis systems that were already put in use, which is facing with the urgent need to improve the handling capacity of the system.Using Hadoop, with MapReduce programming model which is one of its core components, to realize log analysis is the technological trend in recent years. The focus of this study is how to realize distributed log analysis system, and enhance the performance of numerous log analysis systems that have been in use, capitalizing on EB technical accumulation and the relevant technical points of MapReduce.This paper, firstly, introduced the research background and significance of distributed log analysis system, and the research status and other knowledge of log analysis. Secondly, it combined relevant techniques and strategies of Hadoop framework and MapReduce programming model to conduct a detailed analysis and research into some technical points of this system. Thirdly, it proposed a new implementation of distributed log analysis system, based on the use of project-related operation and maintenance platform technology. Finally, based on the actual CRBT project requirements of log analysis, it designed and implemented the distributed CRBT log analysis system.In order to verify the feasibility of distributed log analysis system and the enhancement of performance, this paper, according to project requirements, achieved the CRBT log analysis system by phases, and conducted the function and performance tests. The test results showed that the CRBT log analysis system achieved the project requirements: distributed system effectively improved the performance of log analysis system. It proved that the system can achieve conversion in the distributed framework with very small development cost, by phasing to realize the CRBT log analysis system.