Concept Drift Based On Subspace Learning For Anomaly Detection

Internet develops a lot in the past decades.We can almost not live without it.As the Internet environment can be very complex,its security has drawn the most attention.An attack or abnormal behavior can always cause an enormous loss.In this background,Intrusion Detection System(IDS)is incorporated into this field to solve these web security problems.And,the instantaneity and scale of web data streams require anomaly detecting method has an efficient detecting algorithm.Data stream mining has become a hot topic in data mining field,in which concept drift is an important method.This paper uses concept drifting method to construct anomaly detection.My work is listed as follow:(1)ARLDA is a subspace based concept drifting method,which uses projection variance and projection cosine to detect concept drift.This method has the advantage that it constructs concept detection while reduces dimensionality of the data stream,which improves the efficiency of detection.But,when the concept drift dose not occur in the current data stream,it will choose the model of previous data stream to do the detection.As the uncertainty of web data streams,this choice can be random,compared with nodetection method.Thus,this paper modify this part to choose the model of the first data steam to conduct detection.(2)When concept drift occurs,ARLDA uses 10 percent of the current data stream to train the model,then do the test for the other,which is not in accord with the real environment.Thus,the model and projection matrix used to classify must be the previous ones.This paper come up with an eigenvalue concept set based concept detecting method,which chooses the detecting model via calculating the distance of the eigenvalue vector.In this frame,a data stream can choose the model of the most similar one in the history,which makes the choice more effective and improves the detecting performance.(3)This paper deploys ntopng which is a web probe in a video web service to conduct real network environment experiment.Comparing with the no-detect method,the eigenvalue concept set based concept detecting method performs better.