Research On The Method Of Mining Association Rules For Security Log

With enterprises' facilities expend sharply and systems become more complex,user operations logs are increasing constantly,which lead to that more novel rules are needed for log audit system.The policy database of audit system should be updated continuously,so that the manager can cope with these security logs and find anomalies.Audit system is not applicable to collaborative analysis,and security administrators need an effective way to mine these security logs,get new rules,and update the policy database.Based on the security log of the system event,a method of mining association rules is designed based on Spark,which can analyze big data and heterogeneous security log,effectively eliminate redundancy of the association rules,and excavate novelty rules.The main research contents of the thesis are listed as follows:(1)Invalid data in the log should be cleaned first.After analyzing the characteristics of the security logs' structure,these logs are standardized to audit and mining association rules.Invalid logs are eliminated and transactions are created by time slice method.These transactions are prepared for mining association rules.(2)Aiming at the unbalance of Parallel-Frequent-Pattern Growth algorithm,an improved load value calculation method and an optimal dynamic grouping strategy are proposed,which improve FP-Growth algorithm.The improved algorithm effectively achieves the load balance of the whole mining process based on Spark,then the mining efficiency is speeded up.(3)A method of eliminating redundancy of the association rules is proposed based on domain knowledge.The thesis improves the distance between rules after studying the redundant rules.The novelty of association rules is defined based on domain knowledge,and then added to the DBSCAN algorithm.The association rules are clustered,and these rules with high novelty are screened out from each cluster.Finally,the time efficiency of the proposed algorithm is higher than that of BSPFP under the same environment,which is between 5.1% and 26.4%.The performance and correctness of the improved method of eliminating redundancy are tested.Compared with the association rules based on directed graph mining algorithm,the improved method in the thesis is approximately between 7.5% and 14.0% optimized in the aspect of redundancy elimination.The correctness is tested,which verify the feasibility of the method.