Research And Application Of Log Correlation Analyze Based On MapReduce

Cloud computing is becoming a hot topic in recent years, which rise the revolution in the usage of IT industry. More and more organizations and people prefer to deploy their applications on the cloud.In contrast with putting more resources on the optimization of virtualization, load balance and storage,the provider put fewer efforts on the platform about log analysis,which therefore brought much potential security problems. The propose of this thesis is to solve the details of log analyzing in the cloud computing based on the existing research, the main research of this thesis can be summarized as belows:(1)Propose a new alert correlation mining algorithm based on event sliding window.There are many physical and virtual hosts in the cloud, each of them generates mass logs which is tremendous after aggregation. The task aiming at how to find correlation sequential patterns and extract attack scenarios is given the first priority. It is inefficient and slow when put existing frequent sequential pattern mining algorithm into out logs, which will bring large sums of useless patterns as well as false patterns. Therefore, this thesis presents a new frequent sequential pattern mining algorithm based on event sliding window, together with realizing parallelization on the MapReduce. It can make up the drawback of existing solution as well as improvement in accuracy and speed.(2)Propose a new attribute correlated DDoS attack filtering model based on the request rhythm.Distributed Denial-of-Service attack(DDoS) is a major threat for cloud environment. It is necessary for firewalls and other defense solutions to promote the capacity on solving the DDoS intrusion on various nodes. There was an existing solution which is based on the confidence filtering by analyzing the flow logs in order to construct a normal profile, therefore it could filter the risk packet by comparing with the normal profile. However, this solution aims at solving DDoS threat packets on the network layer and transport layer, which was not enough in solving HTTP DDoS,which takes up a large part of intrusion packets in cloud environment. Therefore, this thesis presents a new packet filtering method based on the session identification and request rhythm,and realizing parallelization on the MapReduce. It can be a compensation for the existing solution which strengthens the defense in HTTP-DDoS identification.(3)We design and implement a system including log collection, log real-time analyze and off-time analyze and model construction. When the above models deployed into the system, both of them can achieve the expected effect with high practical value.