| With the development of container technology,more and more enterprises use containers to build cloud platforms.The structure of the container cloud platform is complex,and the log data generated by each component and application is large in volume,in various data formats,and scattered on different nodes,so that the value of log data cannot be well exploited.This increases the difficulty of platform development and operation and maintenance.Aiming at the status quo and requirements of the container cloud platform,this thesis conducts research on the analysis of log data in the container cloud platform,and designs and implements an intelligent log analysis system.From log collection to log analysis to the final log display,the log analysis system is divided into log collection module,log storage module,log analysis module and data visualization module.Each module is designed and implemented in detail,and the log collection module and log analysis module are the key parts of this thesis.In view of the complex structure of the container cloud platform and the characteristics of fast start and stop of the container,a log collection module with high reliability,high availability and real-time requirements is implemented based on Promtail and Flume.Moreover,in the process of implementation,improvements and supplements have been made to address the deficiencies of Promtail and Flume,including functional supplements and performance optimization.It mainly includes:(1)Intelligent transmission channel is realized,and the transmission channel can be intelligently selected according to the log type and memory congestion,so as to improve the utilization rate of the channel and reduce the transmission time.(2)Loki-sink has been implemented to improve the ability to transmit data to various storage sides.(3)According to the type of log and the actual environment,tune the parameters.In the log analysis module,this thesis proposes an LSTM-based real-time log anomaly detection algorithm Log KC,which not only covers execution path anomalies,but also considers execution component anomalies,adding component awareness technology.In the design and implementation,the existing technology and mechanism have been improved,mainly including:(1)The log parsing algorithm has been improved,and the log parsing speed has been improved.(2)Introduce the abnormal score mechanism,so that the log system is no longer only subject to the algorithm model,and the user-defined score can be used to trigger the alarm.In order to verify the reliability of the system,we conduct detailed tests on the performance and functions of the system.And the anomaly detection algorithm proposed in this thesis is compared and analyzed for precision rate,recall rate and F1 and other indicators.The results show that the system has good results in collection,storage,and log anomaly detection and processing,and can meet the analysis requirements of logs under the container cloud. |
