The Design And Implementation Of A Real Time Log Processing System Under Cloud Computing Environment

Since all the existing cloud computing platforms have a huge organization and complex architecture, each node and module of the platforms generates great amount of log data which scattered in various node, there are many difficulties in log data processing such as various format of log data, huge data transporting, huge data storage and retrieval, which lead to inefficiently of log data accessing, and fruitful information hided in the log data not be fully mined. To solve these problems, this paper conducted a deep study of real time log data processing under cloud computing environment.The main contributions of this paper are 1), To deal with the complicated physical architecture and logical architecture of cloud platform, a complete, scalable real time log analysis system under cloud computing environment have being proposed in distributed system, with the modules and functions real-time streaming data collection, data storage and index, and real-time data analysis.2), This system can help application operator to retrieval log data from the system efficiently, to manage all kinds of log data effectively, and to quickly understand current system status and operation record from the log data. At the same time, the system will also be able to do some data mining work in the large and diverse log data set, unearthed the information contained in the data set. On one hand, tracking the real-time running status of system, and on the other hand, implemented the system failure prediction function based on machine learning algorithms and models, calculate the prediction of what might happen in the future.3), This system provides a friendly interface for data display visual processing, vividly exhibits the information contained in the data from the system, improves the efficiency of data analysis.