Research And Implementation Of Anomaly Detection System Based On The Fusion Framework Of Microservice And Big Data

With the development of the Internet and information technology,information has shown an explosive growth.The Internet has entered the era of big data.The big data processing platform has a huge positive effect on national governance,business decision-making,and market expansion.However,the whole society are faced with the problems brought about by information security while getting such a huge profit.In this period,the Internet is the main channel for data dissemination,and Web services are the most widely used and convenient services.With the widespread use of Web services,attacks on Web services are also increasing.The traditional security products protect the system from the network layer or the transport layer.Nowadays,in order to overcoming the limitations of traditional security products,protecting the system from the application layer has become an urgent need.This paper will analyze the security status of Web services based on the data from application layer and develop an anomaly detection system to protect Web services.The subject comes from the major science and technology projects in Fujian Province "Research and Development of Key Techniques of Real-time Anomaly Detection and Analysis System for Big Data Platform".Based on the key technologies that the project has studied,we combine the specific tasks and requirements of the project.The basic system architecture of the project was built using the microservice framework and the big data lambda framework.A set of methods for implementing RADA with functions such as anomaly detection,anomaly location,and operation status analysis was studied.We have studied the key algorithms--Distributed Frequent Sequence Mining Algorithm,for implementing anomaly detection mining analysis in RADA system.The RADA system is easy to deploy,easy to expand,and easy to manage.In this paper,the main work includes as follows:1.The RADA system integrates related software technologies used in the microservices framework development system,including Spring Boot,MyBatis,Spring Cloud,as well as the lambda framework for big data analysis and related software technologies,including HDFS,Flume,Kafka,and Spark.2.According to the practical application scenario of the RADA system in this project,a detailed requirement analysis is carried out in this paper.According to the microservice framework and the big data lambda framework,the system architecture of these two frameworks was innovatively constructed and the overall design was carried out.3.The system is designed in detail,and the implementation of the microservice framework and the lamda framework are demonstrated.The design and implementation of the fusion technology in the system are emphasized.4.We propose Spark MFPs based on Spark’s maximum frequent pattern mining algorithm to solve the problem of slow mining of distributed frequent sequences.The algorithm deletes non-frequent sequences when constructing a projection database,and generates a maximum frequent sequence candidate set at the same time when recursively searching for deep frequent patterns.Experiments on the test set have demonstrated that SparkMFPs can efficiently mine frequent sequences.This algorithm is the key algorithm in this project.In summary,the fusion framework proposed in this paper based on the microservice framework and the big data lambda framework technology provides a useful reference for the realization of anomaly detection system based on big data technology.At the same time,the new SparkMFPs algorithm proposed in this paper can effectively perform distributed anomaly detection,providing an important guarantee for the industrialization of the project’s achievements.