Research On Distributed Complex Attack Models Detecting Technology

The development of Internet causes more and more network security threats.One of its urgent problems is how to detect the security threats in a real-time and accurate manner.The alert-correlation-based security threats detection technique is becoming the research hotspot.This is because it is coupled with the widely used security products,and it can fully exploit the relation between abnormal events.Security threats is becoming diverse and frequent under the complex and large-scale Internet.This brings two technique challenges.One is that the live alerts tend to be generated with large-scale items,high arrival rates and dynamic.It requires high scalability of alert-correlation-based security threats detection technique,such that the live alerts timely processed.The other is that security threats tend to be updated with unknown types.It requires high activity of alert-correlation-based security threats detection technique,such that the live alerts unknown types can be detected.Existing alert correlation algorithm are mainly focusing on centralized method and have low correlating throughput.Secondly,Existing alert correlation algorithm are based on expert knowledge and cannot detect unknown types.To achieve general scalable and active alert-correlation-based security threats detection technique,this dissertation deeply studies the techniques for known threat types and unknown threat types,respectively.The high velocity alerts data stream requires high scalability and low cost of alert-correlation-based security threats detection technique.To this end,this dissertation proposes a general and scalable causal based alert correlation algorithm,called CausalAC.Through relaxation matching cause and effect between the alerts,CausalAC can effectively detect various types of known threats,and have a certain ability to detect unknown threats type.Due to a hybrid correlation graph partition technique,CausalAC dispatches large-scale skewed alerts to multiple correlating units to achieve parallel matching service.To adapt to the changing of workloads,CausalAC uses a waiting-latency-aware based task scheduling algorithm to ensure load balance.Experiments show that CausalAC has a linear increasing matching capacity as the number of correlating units.Compared with other algorithms,CausalAC has higher throughput by 41% ~ 66%.It can reduce the memory overhead by 65%,and the communication overhead by 12%.The unknown threat type requires high capacity to exploit unknown relation of alert-correlation-based security threats detection technique.To this end,this dissertation proposes a scalable and active data mining based alert correlation algorithm,called ActiveAC.ActiveAC extended Bayesian probability formula with the concept influence-point to calculate the causal correlating probability between the alert types.Through a correlated features set mining technique,ActiveAC exploit the matching features with improved frequent items mining Apriori algorithm.ActiveAC update the influence-point and correlated features set according to change of alert type distribution to reduce computing cost.Meanwhile,ActiveAC uses a flow-aware based task scheduling algorithm to ensure load balance.Experiments show that ActiveAC has a linear increasing matching capacity as the number of correlating units.ActiveAC has a high accuracy with false positive rate less than 5% and false negative rate less than 3%.Based on above theoretical research results,a general and scalable complex threat detection system called GSCTD is designed and implemented on Storm.GSCTD adopts a “dispatch-aggregate” scheme based online alert correlation framework.Based on this framework,we develop passive defense module according to CausalAC and active discover module according to ActiveAC.Passive defense module provide real-time correlating service based on rule base knowledge and active discover module exploit rules to update the rule base.Experiments on various dataset show that GSCTD has a good capacity to detect diverse network threats.For Classical dataset DARPA2000,GSCTD reproduce the DDos attack sequence.For real-world dataset,GSCTD digs out 14 suspicious security threat.Five of them can be judged to be security threats and the other have a significant harm.