Research On Log Anomaly Detection Based On Deep Learning

With the expansion of computer system scale,the order of magnitudes of logs in a fixed amount of time is growing rapidly.Log anomaly can effectively reflect system anomaly,therefore log anomaly detection has become one of the hot research directions recently.Because modern systems generally provide services to users all over the world and operate 24 hours every day,short-term failure of the system may lead to serious consequences,it will cause service suspension of service providers and users.Therefore,the analysis of system faults and business anomalies has become more and more important.It is an significant way to detect system and business exceptions through log exception detection.With the development of deep learning and the improvement of computing power,it becomes technically feasible to use machine learning and neural network for log anomaly detection.Aiming at the problems of complex log system,large amount of log data,various log formats,low accuracy and high false positive rate of existing algorithms,this thesis mainly studies how to solve the problem of anomaly detection with high accuracy from a large amount of log data.A log anomaly detection algorithm based on deep learning and neural network is proposed to realize the functions of system log parsing,encode and anomaly detection.Firstly,this thesis introduces the research background and significance of log anomaly detection algorithm.The related technologies and existing log data analysis schemes and log anomaly detection schemes are systematically studied and investigated.The existing log anomaly detection algorithms are studied.Using the provincial mobile log data set provided by the laboratory project,the log anomaly detection algorithm based on deep learning is realized and improved.The structure of the algorithm and the specific implementation of each module are introduced.The main contributions are as follows:(1)In the process of log parsing,spell algorithm is improved for log data set.Because when there are too many parameters in the log entries,there will be the problem of misjudging the log parameters as log keys.In this thesis,we use periodic splitting and merging to clean up the incorrectly parsed log keys.And in order to train log messages better,replace each word as a token with each logcontent(the effective content of each log)as a token.(2)In the process of implementing the log anomaly detection algorithm,it is found that the existing algorithm has the problem of gradient disappearance.In order to solve this problem,this thesis replaces the LSTM network in the existing Deeplog algorithm with the bidirectional LSTM network,and uses the Adam gradient descent optimization algorithm in the gradient reverse transmission,which has achieved good experimental results,the accuracy of log exception detection has been improved to a certain extent.(3)Based on the improved algorithm proposed in(2),it is found that the anomaly detection algorithm still has the problem of poor generalization ability.In order to improve this problem,we consulted a large number of materials and papers,and found that Adam optimization algorithm has the defect of poor generalization ability through experiments,so Adam optimization algorithm is improved,The learning rate with L2 weight attenuation is used to normalize the weight vector,and only its direction is optimized,so that the learning rate can be accurately controlled,which further improves the generalization performance of the algorithm in the log anomaly detection task.After experimental verification,the accuracy of log detection has been further improved.(4)According to the log anomaly detection algorithm proposed in this thesis,we choose to use the flask framework and docker container technology to complete the construction of the log anomaly detection system after investigation,and send a request to the server through the get interface and post interface to call the log anomaly detection algorithm.And deployed in the server,tested and verified the availability and reliability of the system.