Research On Log Anomaly Detection Technology Based On Behavioral Association Mining

Log analysis is a common means in security audit work.Logs contain rich information and security managers can identify abnormal information by means of log analysis.But existing log anomaly detection methods have the following problems: 1)Logs generated by different systems usually have different formats and it is difficult to refine a common template for anomaly detection;2)Log syntax is more complex.The syntactic patterns of logs for different action behaviors are significantly different.Identifying and grouping different actions into the same attack is a multimodal identification problem.Traditional intrusion detection systems can only detect attacks by simple feature comparison and cannot detect complex attack events consisting of a series of attacks.For the problem of anomaly detection of complex attacks,security managers need to focus not only on the user behavior in the logs,but also to analyze the correlation of user behavior.The advantage of using structured graphical data for anomaly detection is that it can fully explore the correlation in log information and is highly interpretable.There are two application scenarios for anomaly detection through log analysis of user behavior: historical audit and realtime detection.In the historical audit scenario,the problem faced by researchers is how to sufficiently mine the correlations of user behavior in the logs to achieve comprehensive anomaly detection;In the real-time detection scenario,the logs generated in real time cannot form complete correlation relationships.So how to complete efficient and accurate anomaly detection by analyzing the incomplete correlation relationships between user behaviors has become an urgent problem.The main work and innovation points of this thesis are as follows:1)In response to the problem that traditional log security analysis and detection methods cannot detect complex attack events,this thesis proposes an anomaly detection algorithm based on process-time multimodal control flow graphs.The control flow graph represents the flow of user actions,and the process time represents the time interval between actions.This thesis improves the coverage of anomaly detection by adding the property of process time to the traditional control flow graph.The algorithm consists of two parts,pre-processing of log information and composing a control flow graph.In the log information pre-processing link,this thesis proposes a density-based algorithm for discovering clusters to remove the influence of redundant log information on the generation of control flow graphs,eliminate log information of irrelevant user behavior,and improve the efficiency of anomaly detection;In the composition of the control flow graph link,this thesis adopts a multimodal clustering method to merge the attack behaviors belonging to the same attack event,which improves the accuracy of anomaly detection.And experiments show that the anomaly detection accuracy of this algorithm can reach 98.2%.2)In the real-time detection process,the log information generated in real time contains only part of the user behavior actions,and the existing algorithm must ensure that the complete user behavior is obtained before analysis and anomaly detection can be performed.In order to achieve real-time anomaly detection,this thesis proposes a real-time anomaly detection algorithm based on streaming heterogeneous graphs.Streaming refers to processing one edge at a time in the analysis process,and heterogeneous graphs contain nodes and edges with different attributes,and anomaly detection is achieved by matching the similarity of some information in the graph(e.g.,information of some edges).In the stage of similar user behavior matching and anomalous behavior clustering,this thesis proposes a fast anomaly detection algorithm based on Locality Sensitive Hashing(LSH),which is characterized by the ability to convert similar graphs into hash vectors to achieve fast matching of similar subgraphs and improve the efficiency of anomaly detection,and at the end,it is experimentally demonstrated that the anomaly detection algorithm based on streaming heterogeneous graphs proposed in this thesis can perform anomaly detection in a real-time environment.3)For Web service application scenarios,this thesis implements a Web log anomaly detection system based on the above method.The log anomaly detection system is implemented based on the microservice framework and contains the applications and functions of three submodules: log storage module,log analysis module and Web management module.Among them,the log analysis module realizes the anomaly detection in historical audit scenarios and realtime detection scenarios,and the Web management module is used to display the whole process from inputting the log files to be detected to the final display of the anomaly information.This section discusses the main data structures,software architecture design,functional logic design,data structure design,and interface data involved in the system implementation,and gives the results of performance and functional tests in a running environment.