Research And Application Of Template Extraction And Anomaly Detection Based On Log Information

In modern software and hardware systems,logs are an important component that records the system’s operating status,fault information,error logs,etc.With the expansion of computer system scale and the increase in complexity,the number and size of log files are also constantly increasing,posing great challenges to log analysis and processing.Log template parsing and log anomaly detection are two important research areas in log analysis.In the past decades,many log parsers and anomaly detection methods have been proposed.However,the existing methods rely too much on the regular expressions of domain knowledge,ignoring the semantic information of log messages.In addition,the logs of large network service systems are usually multi-source and heterogeneous,which makes the performance of current log parsing and anomaly detection unsatisfactory.Based on this,the thesis focuses on log parsing and log anomaly detection in log analysis,and the main research content includes the following three aspects:(1)The first step in automatic log analysis is log parsing,which converts unstructured raw log messages into structured data.Although the previous log parsing methods have achieved some success,the previous methods rely heavily on specially designed regular expressions and ignore the Semantic information of logs.In order to solve these problems,in this thesis,an automatic log parsing method SNNLog was proposed.The log parsing mechanism combines twin neural networks and fixed parse trees.The fixed parse tree is used as the infrastructure and twin neural networks are used to determine the similarity,this method not only combines the rule model,but also learns the semantic information of log messages,and optimizes the model performance through further post-processing at the later stage of model construction.The experimental results show that SNNLog achieved a F1-score of 0.999 in five datasets,with the highest parsing accuracy in four of datasets.(2)In view of the shortcoming that the previous log anomaly detection methods can not take into account the semantic information of the learning log template and the raw log,this thesis proposes a universal anomaly detection mechanism for heterogeneous logs Log Pal.Log Pal filters the raw system log,then uses SNNLog method to parse the log template,and then combines the template with the raw log to generate log mode events,so as to achieve automatic parsing of heterogeneous logs.Combined with natural language processing and deep learning methods,the Transformer model can be improved to sense the learning log pattern more adaptively and effectively,so as to realize the anomaly detection of heterogeneous logs.The experimental results show that Log Pal can adapt to changes in log types,balance precision and recall,and improve precision,recall,and F1-score on publicly available datasets.(3)Based on the above research content,this thesis finally designs and develops a heterogeneous log intelligent analysis and anomaly detection system based on the proposed log template parsing and anomaly detection mechanism.Firstly,a log template extraction interface is constructed according to the system requirements,which is usually the first step in subsequent log analysis work.Then,a log anomaly detection interface is designed and developed based on the SNNLog method,Finally,based on the above,the functions of log template extraction,log anomaly detection,and log analysis statistics were implemented.