Research On Log Anomaly Detection Technology Based On Deep Analysis

Anomaly detection is vital in solving system failures and defending against network attacks.The log continuously records the dynamic running information of the system,which can reflect the operation status of the system originally,and is of great significance for the monitoring,management and troubleshooting of the system.Therefore,log-based anomaly detection,which aims to discover abnormal behavior and status of the system,has become an important means to ensure the reliability and quality of service systems.However,with the scale of modern services,the system behavior and status are complicated,resulting in an explosive increase in the number of logs.Therefore,manually detecting anomalies for logs becomes difficult to apply due to the limited detection efficiency.Although the traditional machine learning methods have improved the detection efficiency of anomaly detection systems to a certain extent,it is difficult for them to accurately mine the increasingly complex system behavior and state patterns due to the limitation of simple pattern mining.The log anomaly detection method based on deep neural network can not only process massive log data,but also automatically mine complex internal patterns of log data,which is the mainstream research method in the field of log anomaly detection.To this end,we use the deep neural network model as the detection engine,and proposes a log anomaly detection framework PyqLog based on deep analysis.Our work is as follows:(1)A log parsing tool PyqParser based on parse tree search algorithm is proposed.Considering the textualization of log messages,the tool first builds a comprehensive dictionary GDict based on the N-Gram model using 14 kinds of multi-source heterogeneous log datasets.The comprehensive dictionary GDict can improve the generalization ability of PyqParser by constructing n-gram library of log language,so as to deal with the problem of decreasing parsing accuracy caused by iterative evolution of log sentences.Then,in order to improve the efficiency of log parsing,PyqParser based on the idea of fixed depth parsing tree performs n-gram matching by fast search during online parsing to realize the separation of static templates and dynamic parameters.(2)A framework PyqLog is proposed to construct log dynamic events based on the log template and dynamic parameters for log anomaly detection.Different from the existing methods that mainly use log template to construct event sequences to detect abnormal execution path of the system,with the help of information encoding,PyqLog constructs the log template and dynamic parameters as the dynamic log event,considering the information entropy value of static templates and dynamic parameters.Based on the idea that the log dynamic event sequence is regarded as the text sequence of natural language for context analysis,PyqLog can detect the abnormal behavior and state of the system simultaneously by mining the internal pattern of the log sequence.(3)Several groups of comparative experiments were designed and implemented.After sorting out the research ideas of log parsing tools(7 mainstream parsing tools)and log anomaly detection methods(6 traditional machine learning methods and 6 deep learning methods),comparative experiments are designed from multiple evaluation dimensions.On the one hand,it verifies the applicability of our method,and on the other hand,it also shows the performance differences of the mainstream methods.