Design And Implementation Of Huawei's Big Data-Based APT Defense System

In recent years,with the increasing innovation of many emerging industry technolo-gies,worm-type infection attacks have begun to spread on the Internet,and the repre-sentative of such attacks is APT(Advanced Persistent Threat),which is an advanced persistent threat.Traditional security defenses are not ideal for detecting and defend-ing APT.The old detection techniques and methods often take care of each other in terms of defense,resulting in an increase in the number of APT attacks and the losses caused.In order to solve the problem of defending APT more effectively,Huawei has developed a big data-based APT defense system to provide technical support against APT by using popular big data technology.This thesis first studies and summarizes the current status and measures of defense APT at home and abroad,and then analyzes the distributed storage file system HDFS and database Hbase,and studies the theoretical method combining data storage and threat processing technology based on big data scale.Flume,an efficient data collection tool,and Kafka,a news distribution system based on subscription publishing,explored a technical solution for threat detection using Spark Streaming as a core processing tool.Based on the above research,aiming at the persistence and unknownness of APT,this thesis designs and implements an APT defense system based on big data.By using this system,users can get rid of the trouble that traditional security equipment can't track and detect for a long time,and solve the problem of monitoring lag.At the same time,massive storage volume of large data and distributed clustering method also make blind spots of security holes in the past trust zone.The problem is solved,thus avoiding the loss of core asset information caused by APT attacks.This thesis details the overall system development process from requirements analysis,overall structural design to the design of each module and its implementation.The module design part is described in five aspects:acquisition module,distribution module,storage module,detection module and display module.The specific imple-mentation part describes the core functions of the five modules in detail,and gives the relevant implementation.Screenshot screen.Finally,the status quo of the system operation is summarized and forecasted.At present,the official version of the defense system has been put into commercial use.It has been well received by the industry for more than a year,and the social feedback is good.