Research And Implementation Of Web Log Collection And Analysis System Based On ELK

With development of the Internet,the security of network has become an important problem for Internet practitioners.Methods of Web attack are increasing,which have serious impact on enterprises and individuals.The web log security method is an important method of web security analyze.This method can restore the attack scene after the network attack,continue to discover the illegal behavior of users,more convenient for tracking afterwards,and provide effective support for the formulation of security policy.VGG 19 convolutional neural network is studied in this thesis,and the improved VGG19 model is proposed.Then,an anomaly detection model based on the improved VGG 19 convolutional neural network is put forward,and the log collection and analysis system based on ELK framework is developed.Finally,the experimental network with five nodes is established based on Virtual Box,and test of the system is completed.The contents as follows:Firstly,the development of Internet and web security are introduced in this thesis,and the current situation of web security at domestic and abroad is studied.Secondly,the technical and theoretical basis of this thesis are described,including types of Web logs,types of Web attacks,ELK log framework and common methods of log analysis.Thirdly,the establishment process of anomaly detection model is described in detail.(1)The VGG 19 algorithm is improved,and a new anomaly detection model is proposed based on the improved VGG 19.Based on micro migration learning,the convolution layer parameters on image net model are transferred to this model.The number of full connection layers is changed to 1.Softmax Layer has two classification,which improves the training efficiency and accuracy,and makes this model more suitable for network request detection.(2)Based on the ISCX2012 network security data set,the model is trained and achieved good results.Fourthly,the web log collection and analysis system is designed,including the goal,architecture,function module and database.The system includes four modules: log test,log collection,log storage and log analysis result visualization.Finally,the ELK framework is used to realize the web log collection and analysis system.Among them,the log test module mainly analyzes and marks the collected requests through the anomaly detection model proposed in this thesis.The implementation of the three modules,i.e.,log collection,log storage and log analisis result visualization,is based on three open-source tools,including elasticsearch,logstash and kibana.Logstash is used to collect logs;Elasticsearch is used to build a 3-node distributed cluster to store logs;Kibana is used to visualize logs analysis result.At the same time,through the establishment of the experimental network,the log collection and analyze are deployed on the server side,and the normal and abnormal requests are sent to one server node through four nodes,the purpose is to test the anomaly detection of the system.The simulation test results show that the developed system has achieved the expected results.