Research And Application Of Association Mining Algorithm In IDS In Hadoop Framework

With the rapid development of Internet technology,data in terabytes or more will be generated on the network every day.This has caused traditional data mining to be challenged,and big data technology should therefore be born.The emerging discipline of big data has extremely obvious characteristics.First,its enormous capacity and extremely high speed of operation and analysis make traditional data unmatched.Secondly,the high value and diversity of its data is beyond the scope of traditional data.At the same time,regarding the Internet security,the intrusion detection system we generally use only performs the comparison between the acquired data and the existing attack pattern database.This model is better for the types of attacks already existing in the pattern database.However,there is nothing that can be done with new types of attacks or variants of existing attacks.Therefore,we need to further develop and adapt the existing systems to the current ones in order to solve the problems caused by the inability of existing intrusion detection systems to meet the rapid increase in the number of intrusion detection system logs.Due to the above problems,this paper has conducted in-depth research on the application of big data intrusion detection systems.By analyzing the existing flawed frequent item set mining algorithms,we mainly find and improve the optimization algorithms under the existing distributed conditions.This paper selects the relatively perfect Hadoop distributed platform as technical support,and proposes perfection.A new concept of distributed association rule algorithm.First of all,when using multiple computers to work together in big data processing,its processing mode and processing efficiency have always been a difficult problem.MapReduce computing model is used to implement frequent itemsets in the cloud environment using new algorithms.The excavation can do this work more concisely and efficiently.Secondly,adopting the parallel frequent itemsets mining algorithm in Snort intrusion detection system can make upfor the problem that Snort can't make accurate judgments on security events,and also solve the problem that data processing is slow due to large amount of data.The system can determine and distinguish network behaviors by pre-checking modules,and screen out different behaviors from normal ones,thus greatly improving detection efficiency.Its powerful rule dynamic generation module can create new rules in real time against intruded data.The ability to detect intrusions has been greatly improved,and the function of Snort-IDS has been greatly improved and improved.